Initial commit: RAG pipeline for semantic search over personal journal archive
Vector search with cross-encoder re-ranking, hybrid BM25+vector retrieval, incremental index updates, and multiple LLM backends (Ollama local, OpenAI API).
This commit is contained in:
Eric
commit
e9fc99ddc6
43 changed files with 7349 additions and 0 deletions
174
saved_output/2025_08_28.txt
Normal file
174
saved_output/2025_08_28.txt
Normal file
|
|
@ -0,0 +1,174 @@
|
|||
Enter your query (or type 'exit' to quit): I'm looking for the happiest and most joyful passages.
|
||||
Loading llama_index.core.storage.kvstore.simple_kvstore from ./storage_exp/docstore.json.
|
||||
Loading llama_index.core.storage.kvstore.simple_kvstore from ./storage_exp/index_store.json.
|
||||
|
||||
Response:
|
||||
|
||||
**Summary Theme:**
|
||||
The author reflects on moments of joy and happiness in their life, exploring themes such as contentment, love, and the beauty of
|
||||
everyday experiences. They express a desire to let themselves be happy every day and find pleasure in creative pursuits like
|
||||
poetry and art appreciation. Despite personal struggles with depression and anxiety, the author emphasizes the importance of
|
||||
finding happiness in one's daily life.
|
||||
|
||||
**Matching Files:**
|
||||
|
||||
1. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2015-12-03.txt** — Chloe's smile while praising her
|
||||
piano playing brings joy, highlighting the author's appreciation for small acts of kindness.
|
||||
2. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2006-08-11.txt** — The author's day feeding carrots to
|
||||
horses and making pizza with Matthew is described as "fun times, maybe the best ever," showcasing their ability to find joy in
|
||||
simple pleasures.
|
||||
3. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2015-01-23.txt** — Reading poetry and appreciating
|
||||
simple observations brings positive thoughts, indicating a focus on finding happiness through creative pursuits.
|
||||
4. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2024-07-25.txt** — The author reflects on the joys of
|
||||
their life, such as time with family and the love they experienced with T, despite later experiencing heartache.
|
||||
5. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2013-02-15.txt** — The passage encourages being joyful,
|
||||
happy, pleased, and glad, aligning with the author's overall theme of finding happiness in various life experiences.
|
||||
6. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2019-09-14.txt** — Reflecting on a week of learning,
|
||||
teaching, and feeling curious leads to the realization that one can find happiness every day, emphasizing the author's ability to
|
||||
let themselves be happy.
|
||||
7. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2012-09-20.txt** — This file, titled "Ευδαιμονía,"
|
||||
contains ancient Greek words related to happiness and well-being, further reinforcing the author's exploration of finding joy in
|
||||
life.
|
||||
8. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-07-29.txt** — The scorching weather and being
|
||||
outside provide a backdrop to the author's ability to find happiness despite potential physical discomfort, demonstrating their
|
||||
resilient outlook.
|
||||
9. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2024-10-09.txt** — The author expresses frustration and
|
||||
depression due to daily interactions but also acknowledges the importance of finding happiness in life, aligning with their
|
||||
broader theme.
|
||||
10. **file_path: /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2008-12-06.txt** — The passage defines happiness as
|
||||
contentment and peacefulness, highlighting the author's pursuit of a joyful life through their experiences.
|
||||
|
||||
Source documents:
|
||||
2025-07-29.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-07-29.txt 0.7135682886000794
|
||||
2008-12-06.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2008-12-06.txt 0.7099131243276414
|
||||
2009-06-04.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2009-06-04.txt 0.6973211899243362
|
||||
2025-08-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt 0.6866097119060084
|
||||
2013-02-15.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2013-02-15.txt 0.686259123672228
|
||||
2012-09-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2012-09-20.txt 0.6790148415972938
|
||||
2015-01-23.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2015-01-23.txt 0.6761073066656899
|
||||
2015-12-03.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2015-12-03.txt 0.6712531329880593
|
||||
2006-08-11.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2006-08-11.txt 0.6613670040827223
|
||||
2024-07-25.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2024-07-25.txt 0.6570111677987235
|
||||
2025-08-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt 0.6558116128405127
|
||||
2019-09-14.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2019-09-14.txt 0.6549423349658567
|
||||
2024-04-03.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2024-04-03.txt 0.6546862471469852
|
||||
2023-07-24.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2023-07-24.txt 0.6544076938168284
|
||||
2025-08-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt 0.6541587448214657
|
||||
Query processed in 73 seconds.
|
||||
|
||||
|
||||
---
|
||||
|
||||
This was a strange failure!
|
||||
|
||||
((.venv) ) ~/Library/CloudStorage/Dropbox/nd/ssearch/$ run_query.sh
|
||||
Enter your query (or type 'exit' to quit): Find documents that express feelings of gratitude.
|
||||
Loading llama_index.core.storage.kvstore.simple_kvstore from ./storage_exp/docstore.json.
|
||||
Loading llama_index.core.storage.kvstore.simple_kvstore from ./storage_exp/index_store.json.
|
||||
|
||||
Response:
|
||||
|
||||
**Summary Theme:**
|
||||
The query is about finding documents expressing feelings of gratitude. However, it seems there was an error in my interpretation
|
||||
or the context provided, as the dominant themes I identified earlier were related to depression and anxiety rather than gratitude.
|
||||
Based on the given context, the theme that matches the query is related to personal struggles with mental health, particularly
|
||||
feelings of sadness and appreciation for connections.
|
||||
|
||||
**Matching Files:**
|
||||
|
||||
1. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2024-10-09.txt** — Expressed frustration with joggers on the bike
|
||||
path but did not mention gratitude.
|
||||
2. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt** — No direct expressions of gratitude found, but a
|
||||
reflection on personal struggles and achievements was present.
|
||||
3. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2008-05-27.txt** — Focuses on negative emotions like anxiety and
|
||||
anger, with no clear expressions of gratitude.
|
||||
4. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2013-05-23.txt** — Mentions the joy of helping others achieve their
|
||||
goals, which could be interpreted as a form of appreciation or gratitude for their success and recognition.
|
||||
5. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2015-03-17.txt** — Contains suicidal thoughts and negative
|
||||
feelings, indicating a lack of gratitude.
|
||||
6. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2023-07-16.txt** — Describes feelings of loss and the search for
|
||||
meaning, devoid of expressions of gratitude.
|
||||
7. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-03-18.txt** — No clear mentions of gratitude found.
|
||||
8. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2007-05-31.txt** — Focuses on career concerns and negative
|
||||
emotions, without expressing gratitude.
|
||||
9. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2020-05-13.txt** — Struggles with recognizing others' efforts due
|
||||
to internal bad feelings, which contrasts the idea of gratitude.
|
||||
10. **/Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2020-02-01.txt** — Mentions reconnecting with old friendships and
|
||||
family, but there are no explicit expressions of gratitude.
|
||||
|
||||
Source documents:
|
||||
2025-08-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt 0.6865291287082457
|
||||
2008-05-27.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2008-05-27.txt 0.6707430757786356
|
||||
2023-02-17.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2023-02-17.txt 0.6624994985797085
|
||||
2025-08-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt 0.6614406157945066
|
||||
2025-03-18.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-03-18.txt 0.6589271548285772
|
||||
2025-08-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt 0.6583888795181797
|
||||
2025-07-28.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-07-28.txt 0.6575634356770015
|
||||
2012-09-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2012-09-20.txt 0.6564913212073614
|
||||
2020-05-13.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2020-05-13.txt 0.6563809376620068
|
||||
2025-08-20.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-08-20.txt 0.6549296468531686
|
||||
2013-05-23.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2013-05-23.txt 0.653871795081564
|
||||
2009-06-04.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2009-06-04.txt 0.6535844277567499
|
||||
2007-05-31.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2007-05-31.txt 0.6524713123412845
|
||||
2025-07-29.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2025-07-29.txt 0.6517446358739963
|
||||
2020-02-01.txt /Users/furst/Library/CloudStorage/Dropbox/nd/ssearch/data/2020-02-01.txt 0.6514433384900066
|
||||
Query processed in 74 seconds.
|
||||
|
||||
---
|
||||
|
||||
I implemented a regex that srtips the full path:
|
||||
|
||||
((.venv) ) ~/Library/CloudStorage/Dropbox/nd/ssearch/$ run_query.sh
|
||||
Enter your query (or type 'exit' to quit): Entries that discuss testing one's limits, especially emotional and mental.
|
||||
Loading llama_index.core.storage.kvstore.simple_kvstore from ./storage_exp/docstore.json.
|
||||
Loading llama_index.core.storage.kvstore.simple_kvstore from ./storage_exp/index_store.json.
|
||||
|
||||
Response:
|
||||
|
||||
**Summary Theme:**
|
||||
The dominant theme in this context appears to be an individual exploring their emotions, particularly their mental and emotional
|
||||
boundaries, as well as the impact of societal perceptions on feelings. The writer grapples with anxiety, depression, self-worth,
|
||||
and the fear of inadequacy while also contemplating their own mortality and purpose. They seek to understand and manage their
|
||||
emotions, often viewing them as data or information that can guide survival and informed decision-making.
|
||||
|
||||
**Matching Files:**
|
||||
|
||||
1. **file_path: ./data/2023-07-16.txt** — Describes the struggle of wrestling with depression for years, emphasizing the search
|
||||
for meaning in a world driven by efficiency and optimization.
|
||||
2. **file_path: ./data/2015-03-17.txt** — Mentions suicidal thoughts and feeling overwhelmed by negative emotions, indicating a
|
||||
desire to test one's limits emotionally.
|
||||
3. **file_path: ./data/2019-01-14.txt** — Discusses the struggle with controlling impulses and feelings of stress, anxiety, and
|
||||
depression while questioning if one is a prisoner of their biology.
|
||||
4. **file_path: ./data/2025-06-17.txt** — Explores the concept of feeling out personal boundaries and accepting dissonance, which
|
||||
could be seen as testing emotional limits.
|
||||
5. **file_path: ./data/2025-08-20.txt** — Mentions the interest in anarchy while being invested in capital markets and holding a
|
||||
tenured position, indicating a potential exploration of one's limits.
|
||||
6. **file_path: ./data/2017-12-06.txt** — Expresses suicidal thoughts due to burnout and emotional exhaustion, suggesting an
|
||||
attempt to test personal boundaries.
|
||||
7. **file_path: ./data/2017-12-16.txt** — Explores the desire to be a better person and the struggle with balance, potentially
|
||||
indicating a journey of testing one's limits.
|
||||
8. **file_path: ./data/2017-04-13.txt** — Focuses on worrying about hypotheticals and imagined fights, suggesting an exploration
|
||||
of personal boundaries and emotional limits.
|
||||
9. **file_path: ./data/2024-09-20.txt** — Admitted to having depressive thoughts despite appearing jovial, indicating a discussion
|
||||
on testing the limits of one's mental health.
|
||||
10. **file_path: ./data/2025-08-20.txt** — The computer facilitates artistic innovation by freeing the artist from conventional
|
||||
"mental ready-mades," enabling the production of new assemblages of shapes and colors.
|
||||
|
||||
Source documents:
|
||||
2019-01-28.txt ./data/2019-01-28.txt 0.7091032318236316
|
||||
2003-03-09.txt ./data/2003-03-09.txt 0.6819464422399241
|
||||
2025-08-20.txt ./data/2025-08-20.txt 0.6796124657599102
|
||||
2025-08-20.txt ./data/2025-08-20.txt 0.6785008440538487
|
||||
2017-04-13.txt ./data/2017-04-13.txt 0.6768340197245936
|
||||
2022-05-06.txt ./data/2022-05-06.txt 0.6750801120630013
|
||||
2023-01-27.txt ./data/2023-01-27.txt 0.6703347559624786
|
||||
2023-03-14.txt ./data/2023-03-14.txt 0.668340287632692
|
||||
2025-06-17.txt ./data/2025-06-17.txt 0.6656929175939117
|
||||
2025-08-20.txt ./data/2025-08-20.txt 0.6645024849162311
|
||||
2023-07-16.txt ./data/2023-07-16.txt 0.6618312766890652
|
||||
2021-04-15.txt ./data/2021-04-15.txt 0.661171288633267
|
||||
2025-08-20.txt ./data/2025-08-20.txt 0.6600615010925119
|
||||
2019-01-14.txt ./data/2019-01-14.txt 0.6563840810491259
|
||||
2025-05-23.txt ./data/2025-05-23.txt 0.6561484407217757
|
||||
Query processed in 79 seconds.
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue