Chat history storage and search

DataMind Oda keeps every chat as a log of what happened in it, and builds its search from those logs on this computer. This page covers how that store behaves, the limits a search works within and what the cached label in the chat list means; Search earlier chats covers what Oda may look up in your earlier chats.

How your chats are kept

• A chat is kept as an append-only log: Oda adds each new entry and never rewrites what is already there.
• Everything else about a chat, from what the list shows to what a search can find, is derived from its log. The log always leads: a derived copy can lag behind it, but it never contradicts it.
• The logs and the data derived from them stay on this computer. The log files themselves are covered in Chat logs on disk, and downloading a chat as a file is covered in Download a chat log.

The search index

Oda keeps a search index beside the logs, so a search does not have to read every chat.

• The index is derived: Oda builds it from the logs, and searching never changes your chats.
• An installation decides when the index is prepared: on the first search (the default), when Oda starts, or never. With it set to never, a search reports that search is disabled (session search is disabled); reading, exporting and opening your chats still work.
• One running copy of the app owns a given search index at a time. Oda refuses a file that is not a search index it recognises, including the file that holds your chats, and it creates its files so that only your user account can read them.
• To catch up, Oda reads only the chats that are new or changed, prefers a chat's live copy over its stored copy, retries a read that does not settle once, and applies a removal or a replacement as one step, never half of it.
• Inside the app, the store answers whatever search it is asked; the limits that keep a search within the chats it should reach travel with each search, not with the store. Those limits are covered in Search earlier chats.

What a search returns

• A search is a literal phrase, matched as text; it is not a query language.
• Oda ranks the results: more matching text first, then shorter entries, then later time, with ties broken by chat and by the entry's place in it.
• A search returns 20 results by default, never more than 100 at once, and each result carries a snippet of at most 240 characters.
• While it runs, a search pages through results with a reference that stays valid only while the history it searches is unchanged. The same search with its filters in a different order still matches the same reference.

The cached list data

The chat list can answer before a chat is opened, from a saved checkpoint. Values served that way are labelled cached.

• cached values may be as old as the last checkpoint, but they are never wrong; the saved copy catches up as the chat changes.
• Oda saves a checkpoint in the background as the chat changes: about every 200 events or 5 seconds, and always at three moments, when a chat is created, when a turn ends and when the chat closes.
• If a save fails, the last good copy stays and the next save catches up.
• Each saved copy is bound to its chat: it records which log format it was written for, when the chat was created, its folder, and whether the chat began with inherited history. A copy that no longer fits its chat is rebuilt from the log instead of being served.
• There is no setting that clears or limits the saved data, and the saved copy is never rebuilt on a schedule of its own; it is refreshed when Oda reads the chat.

Questions, answered

What does the cached label in the chat list mean?

The chat list can answer from a saved checkpoint instead of loading the chat first, and values served that way are labelled cached. They may be as old as the last checkpoint, but they are never wrong: the chat's log always leads, and the saved copy catches up as the chat changes.

How often does Oda refresh the saved list data?

In the background as the chat changes: about every 200 events or 5 seconds, and always at three moments, when a chat is created, when a turn ends and when the chat closes. A failed save keeps the last good copy, and the next save catches up.

When does Oda prepare the search index?

It depends on the installation: on the first search (the default), when Oda starts, or never. With it set to never, a search reports that search is disabled (session search is disabled), and reading, exporting and opening your chats still work. If you need search and it reports this, ask your administrator; for messages you meet, see Updates and troubleshooting.

Does updating the search index change my chats?

No. The index is derived from the logs. Updating it reads your chats and writes only the index; it never changes what a chat says.

Can search look at a chat that is still open?

Yes. When the index catches up, it reads a chat that is open or still being written from its live copy, so a search sees the chat's current state rather than a stale stored copy.

What happens if new messages arrive while a search runs?

A search pages through results with a reference that fits the history it was searching at that moment. If that history changes while the search runs, the reference goes stale: Oda refuses to continue from an out-of-date place (session-search cursor is stale because its relevant corpus changed) and the search has to be run again from the start, so results are never half old and half new. A reference Oda does not recognise is refused as invalid (session-search cursor is invalid), never guessed.

What happens if a saved copy cannot be read?

Oda never serves a copy it cannot trust: the values are rebuilt from the chat's log instead, and a damaged file is set aside so it cannot hold up the rest. The chat list keeps working, and you may notice nothing.

Can I copy the search data to another computer?

The index is derived data that belongs to one installation: one running copy of the app owns it at a time, and Oda refuses a file it does not recognise as a search index of its own. To move or back up your chats, copy the log folder instead; a new installation builds its own index from the copied logs. See Chat logs on disk.

Can I clear the search index or the saved list data?

No. There is no setting that clears or limits them. Both are derived from your chats and kept in step with them, and the saved copy is never rebuilt on a schedule of its own.