17 · Bucketing

This entry is about a limit nobody looks for until they trip over it. Entries lay flat in the series directory. A dozen files — fine. A hundred and twenty — still fine, though the directory becomes unreadable. Several thousand — and trouble starts, not in the engine and not in the hosting, but in the tools around the repository.

The solution is banal: entries split into subdirectories by year of publication. The change affects only the file layout, because the engine scans the content directory by pattern rather than by a list.

It is worth pausing on the fact that scanning by pattern turned out to be an advantage we had not considered when choosing it. If a list of files existed anywhere and needed updating, this change would have been a rewrite of that list. Since none exists, it was a move of some files.

A more general observation: bottlenecks in systems like this rarely sit where you look for them. We looked in response size and CPU time limits. It turned up in the number of files in a directory — a layer nobody thinks of as part of the architecture.

Hence a rule we now try to apply: for anything that grows linearly with the number of entries, ask what happens at ten thousand. Not in order to optimise immediately — in order to know where the wall is.