Added db load ability (#106)

* Added db load ability, updated README

* Fixed spacing on multiplication

* Fixed double quotes

* Added double quotes to .db

* After reading the code, realized we can return here for .db extension, and let main handle loading the .db into archival memory.
Issue is that the final else was treating the db as a file and reading it in raw, causing issues.

* Removed .csv reference, not currently working.
This commit is contained in:
sk
2023-10-31 15:18:12 -04:00
committed by GitHub
parent e9e5d341a6
commit 9fa75e1752

View File

@@ -152,6 +152,10 @@ def total_bytes(pattern):
def chunk_file(file, tkns_per_chunk=300, model="gpt-4"):
encoding = tiktoken.encoding_for_model(model)
if file.endswith(".db"):
return # can't read the sqlite db this way, will get handled in main.py
with open(file, "r") as f:
if file.endswith(".pdf"):
lines = [l for l in read_pdf_in_chunks(file, tkns_per_chunk * 8)]