From b8647b4c40f71f141308759c6698181fb0b0d70b Mon Sep 17 00:00:00 2001 From: qwO-Owp <332893456+qwO-Owp@users.noreply.github.com> Date: Sun, 18 Oct 2026 20:39:02 +0200 Subject: [PATCH] feat(editor): rank suggestions and make targets by frecency (#140) The ghost, Alt-N/Alt-P and `make ` now prefer what is run often and lately: each distinct line scores the sum of 2^16 >> (age / 64) over its occurrences in the HELLISH_SUGGEST_MAX window, ties to the newest. HELLISH_SUGGEST_STRATEGY=recent restores newest-first; read every prompt. One ranking primitive (rl_suggest_rank) feeds the ghost and the cycle; it groups by sort over a reused scratch buffer instead of libft's t_hash, which strdups every key (per-key malloc churn). The fast path is kept: a score does not depend on the prefix, so a winner that still matches a longer line stays the best of the smaller set. make TAB turns readline's match sort off only for its own answer; every attempt starts with it on. libc qsort, not ft_qsort: ft_qsort pivots on the last element and is O(n^2) on runs of identical entries, which arrive already in ranking order: 6.76 s for one key over 20000 copies (F13 red), under 1.5 s with qsort. Tests: autosuggest F1-F13 plus a randomized model-vs-shell cycle check, case_cycle_dupes pinned to `recent` with a frecent twin (the only existing case whose order changes), make cases 25-29 (order, sort-flag restore, recent, user complete spec, -C/-f), frontend budget with 1000 distinct entries and Backspace: 6.0 syscalls / 1.1 writes per key. Mutants MF1-MF11 (age, frequency, tie, direction, strategy x2, window, sort flag, make history, cycle order, clean filter) each turn cases red. --- USER_DOC.md | 13 +- incs/history.h | 6 + incs/prompt.h | 6 +- incs/shell.h | 1 + src/completion/complete_make.c | 19 +- src/completion/complete_make.h | 25 ++ src/completion/complete_make_cache.c | 1 + src/completion/complete_make_rank.c | 130 ++++++++++ src/completion/complete_make_rank2.c | 80 +++++++ src/completion/complete_provider.c | 8 + src/completion/completion.c | 8 +- src/helpers/free_utils.c | 4 + src/infrastructure/history_frecency.c | 67 ++++++ src/infrastructure/rl_private.h | 12 +- src/platform/posix/rl_slot_env.c | 13 + src/platform/posix/rl_suggest.c | 24 +- src/platform/posix/rl_suggest4.c | 8 +- src/platform/posix/rl_suggest5.c | 81 +++---- src/platform/posix/rl_suggest7.c | 42 +--- src/platform/posix/rl_suggest8.c | 118 ++++++++++ tests/autosuggest_test.py | 326 +++++++++++++++++++++++++- tests/complete_make_test.py | 112 ++++++++- tests/frontend_budget_test.py | 50 +++- wiki/manual.md | 1 + 24 files changed, 1038 insertions(+), 117 deletions(-) create mode 100644 src/completion/complete_make_rank.c create mode 100644 src/completion/complete_make_rank2.c create mode 100644 src/infrastructure/history_frecency.c create mode 100644 src/platform/posix/rl_suggest8.c diff --git a/USER_DOC.md b/USER_DOC.md index 63ff105c..d7c12a3b 100644 --- a/USER_DOC.md +++ b/USER_DOC.md @@ -169,6 +169,10 @@ escape hatches above get you back to your previous shell at any time. current directory (also `gmake`, `/usr/bin/make`, and behind `sudo`, `env X=1`, `&&`, `(`; a wrapper option that takes an argument, as in `sudo -u root make`, falls back to filenames). The file is parsed in the shell itself; make is never run. +Targets you have run (history lines starting with `make` or `gmake`) come first, +most-run (weighted toward recent runs) first; +the rest follow alphabetically. With `HELLISH_SUGGEST_STRATEGY=recent` the +run ones are ordered by how recently they ran. - Files read: the first regular file of `GNUmakefile`, `makefile`, `Makefile` (GNU make's order, symlinks followed), at most 4 MiB. No such file, a @@ -194,8 +198,9 @@ falls back to filenames). The file is parsed in the shell itself; make is never |---|---| | `HELLISH_BANNER=0\|1` | welcome panel off / on | | `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down plain history; default is prefix search, read at every prompt | -| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the newest history line that starts with what you typed, shown dim after the cursor (see below); read at every prompt | +| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the best-ranked history line (see `HELLISH_SUGGEST_STRATEGY`) that starts with what you typed, shown dim after the cursor (see below); read at every prompt | | `HELLISH_SUGGEST_MAX=N` | how many of the newest history lines the suggestion looks at (default 1000; not a number, or below 1, is the default); read at every prompt | +| `HELLISH_SUGGEST_STRATEGY=recent` | rank suggestions newest first; by default (unset or any other value) they are ranked by frecency: how often a line was run, weighted toward recent runs. Also orders `make `. Read at every prompt | | `HELLISH_NO_UPDATE_CHECK=1` | never check for updates, no badge | | `HELLISH_NO_ANIM=1` | no startup animation | | `HELLISH_NO_EXEC=1` | user-install hook: skip for this one session | @@ -219,10 +224,10 @@ typed, and the hint is wiped from the screen. A word is the blanks at the front of the hint plus the run of characters up to the next space or tab (so `-m`, `a/b`, `x=1` are one word each). Alt-N and Alt-P step through distinct lines (same full text offered once), look -at no more than `HELLISH_SUGGEST_MAX` entries, ring the bell at the oldest -and the newest end instead of wrapping, and start over from the newest +at no more than `HELLISH_SUGGEST_MAX` entries, go in ranking order, ring the bell at the last and +the first instead of wrapping, and start over from the best match as soon as the line is edited. Right after Alt-N inserts the line -shown, not the newest. A hint a zle widget set as `POSTDISPLAY` is not +shown, not the best-ranked one. A hint a zle widget set as `POSTDISPLAY` is not cycled or taken by words. In vi insert mode Alt-F / Alt-N / Alt-P work the same (no Ctrl-Right). `unset HELLISH_AUTOSUGGEST` (or `=0`) turns it off at the next prompt. A key you bound yourself diff --git a/incs/history.h b/incs/history.h index 790b8a92..994fb8b8 100644 --- a/incs/history.h +++ b/incs/history.h @@ -15,6 +15,7 @@ # include "shell.h" # include +# include /* hist_cmds is the truth; readline's own list is a mirror kept in step so the arrow keys agree with what `history` prints. @@ -67,5 +68,10 @@ t_string encode_cmd_hist(char *cmd); void manage_history(t_shell *state); bool worthy_of_being_remembered(t_shell *state); void add_history_line(t_shell *state, const char *cmd); +/* Frecency (history_frecency.c). */ +uint32_t hist_weight(size_t age); +uint64_t hist_score_add(uint64_t score, size_t len, size_t idx, + bool recent); +size_t hist_window_floor(size_t len, int max); #endif \ No newline at end of file diff --git a/incs/prompt.h b/incs/prompt.h index 8eafbe4d..802621bc 100644 --- a/incs/prompt.h +++ b/incs/prompt.h @@ -108,8 +108,10 @@ typedef struct s_rl char *sg_post; /* POSTDISPLAY a widget set (zle_rl5.c) */ char *sg_post_at; /* the line it was set for */ char *sg_pub; /* what a widget was handed as POSTDISPLAY */ - t_vec sg_cyc; /* history indices offered (size_t) */ - size_t sg_cyc_k; /* which of them is on screen */ + bool sg_recent; /* HELLISH_SUGGEST_STRATEGY=recent */ + t_vec sg_rank; /* ranked history indices (size_t) */ + t_vec sg_scr; /* ranking scratch (t_sg_ent), reused */ + size_t sg_cyc_k; /* which of sg_rank is on screen */ char *sg_cyc_line; /* line the cycle is for; NULL: none */ bool sg_painted; /* a ghost is on screen right now */ int sg_col; /* column it starts at */ diff --git a/incs/shell.h b/incs/shell.h index 29186cb2..1e2d2d84 100644 --- a/incs/shell.h +++ b/incs/shell.h @@ -332,6 +332,7 @@ typedef struct s_make_cache long long mtime_s; long long mtime_ns; t_vec targets; /* char *, xmalloc'd */ + t_vec order; /* const char *, targets ranked for this TAB */ } t_make_cache; typedef struct s_shell diff --git a/src/completion/complete_make.c b/src/completion/complete_make.c index c9601da2..f0f7cd32 100644 --- a/src/completion/complete_make.c +++ b/src/completion/complete_make.c @@ -34,6 +34,9 @@ ** TAB. LIMITATION: a change to an INCLUDED file alone does not invalidate ** it -- touch the Makefile. ** +** ORDER. Targets used in history come first (frecency, see +** complete_make_rank.c), then the rest alphabetically. +** ** WHEN IT STAYS OUT. A word starting with '-' or holding '=' or '/' is not ** a target; and if an earlier argument is -C, --directory*, -f, --file* or ** --makefile*, the Makefile is not the one in the cwd, so filenames win. @@ -54,12 +57,12 @@ static char *mk_generator(const char *text, int state) t_vec *v; char *s; - v = &(*zle_state_cell())->make_cache.targets; + v = &(*zle_state_cell())->make_cache.order; if (!state) i = 0; while (i < v->len) { - s = ((char **)v->ctx)[i++]; + s = (char *)((const char **)v->ctx)[i++]; if (!ft_strncmp(s, text, ft_strlen(text))) return (rl_dup(s)); } @@ -90,11 +93,21 @@ static int mk_diverts(t_pv_line *a) return (0); } +/* The matches come out of mk_generator already in the order worth + showing (mk_order), so readline must not sort them again: it does that + AFTER this returns, so the flag is cleared here and cmd_completion sets + it back at the start of every attempt (completion.c). */ char **comp_make(t_shell *st, const char *text, t_pv_line *a) { + char **m; + if (text[0] == '-' || ft_strchr(text, '=') || ft_strchr(text, '/')) return (NULL); if (mk_diverts(a) || !mk_load(st)) return (NULL); - return (rl_completion_matches(text, mk_generator)); + mk_order(st); + m = rl_completion_matches(text, mk_generator); + if (m) + rl_sort_completion_matches = 0; + return (m); } diff --git a/src/completion/complete_make.h b/src/completion/complete_make.h index b4cf7865..53aa0109 100644 --- a/src/completion/complete_make.h +++ b/src/completion/complete_make.h @@ -49,6 +49,25 @@ typedef struct s_mk_scan int def; } t_mk_scan; +/* A target while ranking by history: how much it was used, and where. */ +typedef struct s_mk_use +{ + const char *name; + uint64_t score; + size_t newest; + size_t seen; +} t_mk_use; + +/* What counting needs besides the targets: the history length (for age) + and the strategy. */ +typedef struct s_mk_rank +{ + t_mk_use *u; + size_t cnt; + size_t len; + bool recent; +} t_mk_rank; + # define MK_MAX 4194304 /* The nanosecond mtime field is st_mtim on POSIX 2008, st_mtimespec on @@ -62,11 +81,17 @@ typedef struct s_mk_scan size_t pc_cmd_off(int start); size_t prov_word(t_pv_line *p, const char **w); size_t prov_base(const char *w, size_t n); +bool prov_is_make(const char *w, size_t n); char **comp_provider_try(const char *text, int start, int end); char **comp_make(t_shell *st, const char *text, t_pv_line *a); char *rl_dup(const char *s); t_vec *mk_load(t_shell *st); void mk_cache_free(t_shell *st); +void mk_order(t_shell *st); +int mk_cmp_use(const void *x, const void *y); +int mk_cmp_name(const void *x, const void *y); +void mk_count_entry(t_mk_rank *r, const char *e, size_t idx); +bool mk_is_make_entry(const char *e); char *mk_slurp(const char *path, size_t *len); void mk_scan(t_mk_scan *m, const char *buf, size_t len); void mk_line(t_mk_scan *m, char *l); diff --git a/src/completion/complete_make_cache.c b/src/completion/complete_make_cache.c index 26cc3dc7..2def35de 100644 --- a/src/completion/complete_make_cache.c +++ b/src/completion/complete_make_cache.c @@ -44,6 +44,7 @@ void mk_clear(t_make_cache *c) while (i < c->targets.len) xfree(((char **)c->targets.ctx)[i++]); xfree(c->targets.ctx); + xfree(c->order.ctx); xfree(c->name); ft_bzero(c, sizeof(*c)); } diff --git a/src/completion/complete_make_rank.c b/src/completion/complete_make_rank.c new file mode 100644 index 00000000..67d93c43 --- /dev/null +++ b/src/completion/complete_make_rank.c @@ -0,0 +1,130 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* complete_make_rank.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "complete_make.h" +#include + +/* make ordering: the targets the history shows being used come +** first, best frecency first (history_frecency.c, the ranking the inline +** suggestion uses); the rest follow alphabetically. +** +** An entry counts for a target when its FIRST word is a command the +** provider table maps to comp_make (make, gmake, any path to them) and +** the target is one of its whitespace-separated words, once per entry. +** +** Caveat: this reads words, not make's command line. `make -C re test` +** credits a target named `re` for the directory, and in `make a && make b` +** the `b` counts because the entry starts with make: a target can rank +** higher than it deserves, but is never hidden. `sudo make x` is not +** counted, and a word with `;`, a quote or a backslash glued on never +** equals a target name, so those under-credit. HELLISH_SUGGEST_STRATEGY= +** recent, or an alphabetical list, is the way out. */ + +/* Alphabetical, the way readline sorts when it is allowed to (strcoll), + so an unused-only list looks exactly as it always did. */ +int mk_cmp_name(const void *x, const void *y) +{ + const t_mk_use *a; + const t_mk_use *b; + + a = x; + b = y; + return (strcoll(a->name, b->name)); +} + +/* Used targets first: best score, then newest; ties, and every unused + target (score 0), by name. */ +int mk_cmp_use(const void *x, const void *y) +{ + const t_mk_use *a; + const t_mk_use *b; + + a = x; + b = y; + if (a->score != b->score && a->score < b->score) + return (1); + if (a->score != b->score) + return (-1); + if (a->newest != b->newest && a->newest < b->newest) + return (1); + if (a->newest != b->newest) + return (-1); + return (mk_cmp_name(x, y)); +} + +/* Is the entry a make invocation: its first word names the make provider + (the same table row that decides `make ` itself). */ +bool mk_is_make_entry(const char *e) +{ + size_t n; + size_t b; + + while (*e == ' ' || *e == '\t') + e++; + n = 0; + while (e[n] && e[n] != ' ' && e[n] != '\t') + n++; + b = prov_base(e, n); + return (prov_is_make(e + b, n - b)); +} + +/* Binary search of the name-sorted `u` for the word w[0..n); NULL if it + is not a target. */ +static t_mk_use *mk_find(t_mk_use *u, size_t cnt, const char *w, size_t n) +{ + size_t lo; + size_t hi; + size_t mid; + int c; + + lo = 0; + hi = cnt; + while (lo < hi) + { + mid = lo + (hi - lo) / 2; + c = ft_strncmp(u[mid].name, w, n); + if (!c && u[mid].name[n]) + c = 1; + if (!c) + return (&u[mid]); + if (c < 0) + lo = mid + 1; + else + hi = mid; + } + return (NULL); +} + +/* Credit entry `idx` (text `e`) to every distinct target among its words. + `seen` stops a word repeated in one entry from counting twice. */ +void mk_count_entry(t_mk_rank *r, const char *e, size_t idx) +{ + t_mk_use *t; + size_t n; + + while (*e) + { + while (*e == ' ' || *e == '\t') + e++; + n = 0; + while (e[n] && e[n] != ' ' && e[n] != '\t') + n++; + t = mk_find(r->u, r->cnt, e, n); + if (t && n && t->seen != idx + 1) + { + t->seen = idx + 1; + t->score = hist_score_add(t->score, r->len, idx, r->recent); + t->newest = idx; + } + e += n; + } +} diff --git a/src/completion/complete_make_rank2.c b/src/completion/complete_make_rank2.c new file mode 100644 index 00000000..7e8cb98b --- /dev/null +++ b/src/completion/complete_make_rank2.c @@ -0,0 +1,80 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* complete_make_rank2.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "complete_make.h" +#include +#include + +/* Fill st->make_cache.order with the cached targets in the order + `make ` offers them (complete_make_rank.c): used first, by + frecency, then the rest alphabetically. */ + +/* One record per target, sorted by name so that words can be looked up + by binary search. Returns NULL for no targets or no memory. */ +static t_mk_use *mk_records(t_vec *targets) +{ + t_mk_use *u; + size_t i; + + if (!targets->len) + return (NULL); + u = xmalloc(targets->len * sizeof(*u)); + i = 0; + while (i < targets->len) + { + u[i] = (t_mk_use){((char **)targets->ctx)[i], 0, 0, 0}; + i++; + } + qsort(u, targets->len, sizeof(*u), mk_cmp_name); + return (u); +} + +/* Walk the window of the newest HELLISH_SUGGEST_MAX history entries, the + same one the inline suggestion looks at, crediting make invocations. + Cost: O(window * words * log targets) per TAB. */ +static void mk_credit(t_shell *st, t_mk_rank *r) +{ + char **a; + size_t i; + + a = (char **)st->hist.hist_cmds.ctx; + i = hist_window_floor(r->len, st->rl.sg_max); + while (i < r->len) + { + if (a[i] && mk_is_make_entry(a[i])) + mk_count_entry(r, a[i], i); + i++; + } +} + +/* The ranked order, as borrowed pointers into the target list. It is + rebuilt on every TAB: the history moves on, the targets rarely do. */ +void mk_order(t_shell *st) +{ + t_mk_rank r; + size_t i; + + st->make_cache.order.len = 0; + st->make_cache.order.elem_size = sizeof(const char *); + r.cnt = st->make_cache.targets.len; + r.u = mk_records(&st->make_cache.targets); + r.len = st->hist.hist_cmds.len; + r.recent = st->rl.sg_recent; + if (!r.u) + return ; + mk_credit(st, &r); + qsort(r.u, r.cnt, sizeof(*r.u), mk_cmp_use); + i = 0; + while (i < r.cnt) + vec_push(&st->make_cache.order, &r.u[i++].name); + xfree(r.u); +} diff --git a/src/completion/complete_provider.c b/src/completion/complete_provider.c index 270b7e89..8c57493d 100644 --- a/src/completion/complete_provider.c +++ b/src/completion/complete_provider.c @@ -75,6 +75,14 @@ static t_prov_fn prov_find(const char *w, size_t n) return (NULL); } +/* Does the command named by w[0..n) belong to the make provider (make, + gmake)? The table is the one place that says so; the history ranking in + complete_make_rank.c asks instead of keeping a list of its own. */ +bool prov_is_make(const char *w, size_t n) +{ + return (prov_find(w, n) == comp_make); +} + /* Dispatch an argument word to the provider of its command, or NULL. */ char **comp_provider_try(const char *text, int start, int end) { diff --git a/src/completion/completion.c b/src/completion/completion.c index f9025ba4..d285bed4 100644 --- a/src/completion/completion.c +++ b/src/completion/completion.c @@ -101,13 +101,19 @@ char *rl_dup_dollar(const char *name, size_t len) Only when no spec exists does a built-in provider get a turn (make targets, complete_make.c), so a user's `complete` always wins; a provider - that has nothing to offer answers NULL and the filenames come back. */ + that has nothing to offer answers NULL and the filenames come back. + + readline sorts the matches after this returns (postprocess_matches), so + a provider that returns an order of its own (make) can only switch the + sort off for that attempt; the flag is therefore set back on here, for + every attempt, or a later filename or command TAB would stay unsorted. */ static char **cmd_completion(const char *text, int start, int end) { char **spec; rl_completion_append_character = ' '; rl_completion_suppress_append = 0; + rl_sort_completion_matches = 1; if (text[0] == '$' && !ft_strchr(text, '/')) return (rl_attempted_completion_over = 1, complete_variables(text, start, end)); diff --git a/src/helpers/free_utils.c b/src/helpers/free_utils.c index 8953f77a..eac090bc 100644 --- a/src/helpers/free_utils.c +++ b/src/helpers/free_utils.c @@ -112,6 +112,10 @@ static void free_session_strings(t_shell *state) state->dft_ctx = 0; xfree(state->rl.buff.ctx); xfree(state->rl.rp_txt.ctx); + xfree(state->rl.sg_rank.ctx); + xfree(state->rl.sg_scr.ctx); + state->rl.sg_rank = (t_vec){}; + state->rl.sg_scr = (t_vec){}; xfree(state->hist.file); state->hist.file = NULL; if (state->path_dirs) diff --git a/src/infrastructure/history_frecency.c b/src/infrastructure/history_frecency.c new file mode 100644 index 00000000..4cbaa227 --- /dev/null +++ b/src/infrastructure/history_frecency.c @@ -0,0 +1,67 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* history_frecency.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "shell.h" + +/* Frecency: how much a history entry is worth as a suggestion, from how +** often AND how recently its text was run. Shared by the inline ghost +** and Alt-N/P (platform/posix/rl_suggest8.c) and by `make ` +** (completion/complete_make_rank.c), so both rank the same way. +** +** An occurrence's weight starts at 2^16 and halves every FRE_HALF newer +** entries, down to 1 once it is 16 halvings old; a text scores the sum of +** its occurrences. The score does not depend on what is being typed, so +** the best match for a prefix stays the best for every longer prefix it +** still matches, which is what lets the ghost skip rescanning per key. +** +** Caveat: age is a count of entries, not time. A burst of 64 unrelated +** commands halves a habit's weight, while a command last used a year ago +** in a quiet shell still weighs as much as one used yesterday; and +** FRE_HALF = 64 is a guess, not a measurement. HELLISH_SUGGEST_MAX bounds +** the window, so nothing older than it counts at all. Escape hatch: +** HELLISH_SUGGEST_STRATEGY=recent ignores frequency entirely. */ +#define FRE_HALF 64 +#define FRE_STEPS 16 + +/* Weight of an entry `age` entries older than the newest. */ +uint32_t hist_weight(size_t age) +{ + size_t steps; + + steps = age / FRE_HALF; + if (steps > FRE_STEPS) + steps = FRE_STEPS; + return ((uint32_t)(1u << FRE_STEPS) >> steps); +} + +/* Fold the occurrence at index `idx` (of `len` entries) into `score`. + `recent` makes the score the newest index (+1, so 0 stays "unused"), + which orders newest first and collapses repeats. */ +uint64_t hist_score_add(uint64_t score, size_t len, size_t idx, + bool recent) +{ + if (recent) + { + if ((uint64_t)idx + 1 > score) + return ((uint64_t)idx + 1); + return (score); + } + return (score + hist_weight(len - 1 - idx)); +} + +/* First index of the window of the newest `max` entries. */ +size_t hist_window_floor(size_t len, int max) +{ + if (max > 0 && len > (size_t)max) + return (len - (size_t)max); + return (0); +} diff --git a/src/infrastructure/rl_private.h b/src/infrastructure/rl_private.h index ccece657..7fe01086 100644 --- a/src/infrastructure/rl_private.h +++ b/src/infrastructure/rl_private.h @@ -97,6 +97,15 @@ typedef struct s_frame char *rl_suggest_update(t_shell *state, int cols); char *rl_suggest_live(t_rl *l); +/* One candidate while ranking: the text points into hist_cmds and is only + good for the call that filled it. */ +typedef struct s_sg_ent +{ + const char *text; + size_t idx; + uint64_t score; +} t_sg_ent; + char *rl_suggest_find(t_shell *state, t_rl *l); int rl_suggest_accept(int count, int key); t_shell *rl_suggest_ghost_state(void); @@ -104,7 +113,8 @@ int rl_suggest_bell(void); int rl_suggest_word(int count, int key); int rl_suggest_older(int count, int key); int rl_suggest_newer(int count, int key); -ssize_t rl_suggest_scan(t_shell *state, t_rl *l, size_t from); +size_t rl_suggest_cands(t_shell *state, t_rl *l); +size_t rl_suggest_rank(t_shell *state, t_rl *l); void rl_suggest_cyc_clear(t_rl *l); bool rl_suggest_cyc_sync(t_shell *state, t_rl *l); void rl_suggest_end(t_shell *state); diff --git a/src/platform/posix/rl_slot_env.c b/src/platform/posix/rl_slot_env.c index e1375d37..2cfc6a6b 100644 --- a/src/platform/posix/rl_slot_env.c +++ b/src/platform/posix/rl_slot_env.c @@ -64,6 +64,18 @@ static int sg_limit(t_shell *state) return (ft_atoi(e->value)); } +/* HELLISH_SUGGEST_STRATEGY: how suggestions are ranked. Exactly `recent` + is newest first; unset, empty or anything else (typos included) is the + default, frecency, so a misspelt value never silently changes order to + the old one. */ +static bool sg_wants_recent(t_shell *state) +{ + t_env *e; + + e = env_get(&state->env, "HELLISH_SUGGEST_STRATEGY"); + return (e && e->value && !ft_strcmp(e->value, "recent")); +} + /* Interactive reads only, right before rl_preinit: note what is wanted and ** switch the slot hooks on. Doing it inside rl_preinit's caller keeps ** `bind -p` (which also runs rl_preinit, in -c mode) exactly as bash's. @@ -76,4 +88,5 @@ void rl_slot_sync(t_shell *state) state->rl.feat_want[FEAT_HIST] = hp_wanted(state); state->rl.feat_want[FEAT_ACCEPT] = as_wanted(state); state->rl.sg_max = sg_limit(state); + state->rl.sg_recent = sg_wants_recent(state); } diff --git a/src/platform/posix/rl_suggest.c b/src/platform/posix/rl_suggest.c index c74a0a5d..44f61c48 100644 --- a/src/platform/posix/rl_suggest.c +++ b/src/platform/posix/rl_suggest.c @@ -12,7 +12,8 @@ #include "rl_private.h" -/* The inline suggestion: the newest history entry that starts with what +/* The inline suggestion: the best-ranked history entry (frecency, or the +** newest with HELLISH_SUGGEST_STRATEGY=recent) that starts with what ** is typed, shown dim after the cursor and never part of the line. This ** file decides WHAT may be suggested and WHEN; rl_suggest2.c turns it into ** the text and rl_rprompt.c paints it. @@ -23,16 +24,16 @@ ** is a C1 control in its UTF-8 form: running `printf '\e[31m'` once must ** not repaint every later prompt. */ -/* The newest of the last sg_max entries that starts with the first `n` - bytes of the line, is longer than it and is clean (rl_suggest_scan). */ +/* The best-ranked of the last sg_max entries that start with the line, are + longer than it and are clean (rl_suggest_rank), or NULL. */ static char *sg_scan(t_shell *state, t_rl *l) { - ssize_t i; + char **a; - i = rl_suggest_scan(state, l, state->hist.hist_cmds.len); - if (i < 0) + if (!rl_suggest_rank(state, l)) return (NULL); - return (((char **)state->hist.hist_cmds.ctx)[i]); + a = (char **)state->hist.hist_cmds.ctx; + return (a[*(size_t *)l->sg_rank.ctx]); } /* True when the last scan still answers for the line: the line extends the @@ -54,9 +55,12 @@ static bool sg_cached(t_rl *l, size_t n) ** Typing extends the line, and an entry that matches the longer line ** matched the shorter one too -- so when the line still starts with the one ** the last scan was for, the answer is the same entry as long as it still -** matches (or "nothing" for good). That makes a key O(line) with no scan, -** and a paste of 4 KB one scan instead of 4096. Anything else (a -** Backspace, a recalled line) scans again. The history cannot change under +** matches (or "nothing" for good). That holds for the frecency ranking as +** well: an entry's score does not depend on the line, so the winner for +** the shorter line, still matching, is the best of the smaller set too. +** That makes a key O(line) with no scan, and a paste of 4 KB one scan +** instead of 4096. Anything else (a Backspace, a recalled line) scans +** again. The history cannot change under ** it: it only grows between reads, and the cache is dropped per read. */ char *rl_suggest_find(t_shell *state, t_rl *l) { diff --git a/src/platform/posix/rl_suggest4.c b/src/platform/posix/rl_suggest4.c index 1dd645ab..5703af41 100644 --- a/src/platform/posix/rl_suggest4.c +++ b/src/platform/posix/rl_suggest4.c @@ -53,7 +53,9 @@ void rl_suggest_stale(void) /* The read is over (or about to start): nothing of the ghost, of the search it was found with or of a widget's POSTDISPLAY outlives it. The - history may have changed since, so the cache cannot either. */ + history may have changed since, so the cache cannot either. The ranking + buffers are freed here too: they are reused by every key of one read, + never kept between reads. */ void rl_suggest_end(t_shell *state) { t_rl *l; @@ -65,6 +67,10 @@ void rl_suggest_end(t_shell *state) xfree(l->sg_post_at); xfree(l->sg_pub); rl_suggest_cyc_clear(l); + xfree(l->sg_rank.ctx); + xfree(l->sg_scr.ctx); + l->sg_rank = (t_vec){}; + l->sg_scr = (t_vec){}; l->sg_ent = NULL; l->sg_prev = NULL; l->sg_post = NULL; diff --git a/src/platform/posix/rl_suggest5.c b/src/platform/posix/rl_suggest5.c index 484134c1..d59204ee 100644 --- a/src/platform/posix/rl_suggest5.c +++ b/src/platform/posix/rl_suggest5.c @@ -12,11 +12,10 @@ #include "rl_private.h" -/* Scanning the history for suggestions, and the state of Alt-N / Alt-P -** cycling. A cycle remembers the history INDICES it has shown (sg_cyc, a -** stack: Alt-N pushes the next older distinct one, Alt-P pops back), so -** going back to a newer alternative costs nothing and a revisit never -** re-scans. */ +/* Collecting suggestion candidates from the history, and the state of +** Alt-N / Alt-P cycling. A cycle is the ranking (rl_suggest_rank) made +** once, as history INDICES in sg_rank, and a position in it: Alt-N and +** Alt-P only move the position, so nothing is ever scanned again. */ /* True when `s` holds nothing a terminal would interpret. 0x7f is DEL; the pair 0xC2 0x80..0x9F is U+0080..U+009F, which some terminals read as a @@ -37,63 +36,43 @@ static bool sg_clean(const unsigned char *s) return (true); } -/* True when an entry with the same full text was already offered in this - cycle. Distinct means distinct text: history keeps duplicates, and - stepping through three copies of one command is not a choice. */ -static bool sg_offered(t_shell *state, t_rl *l, const char *text) -{ - char **a; - size_t *v; - size_t i; - - a = (char **)state->hist.hist_cmds.ctx; - v = (size_t *)l->sg_cyc.ctx; - i = 0; - while (i < l->sg_cyc.len) - { - if (!ft_strcmp(a[v[i]], text)) - return (true); - i++; - } - return (false); -} - -/* The index of the next older entry below `from` that starts with the -** line, is longer than it, is clean and was not offered yet; -1 if none. -** Only the newest sg_max entries are looked at (HELLISH_SUGGEST_MAX), so a -** huge history cannot slow a key down. +/* Fill the scratch with every candidate of the window of the newest +** sg_max entries: starts with the line, is longer than it, is clean. +** Returns how many. The scratch and its capacity are kept for the next +** call (a key never allocates once it has grown), and the texts point into +** the history, so they are good until the history changes. ** -** Cost: O(sg_max * (line + offered)) per call, offered being the entries -** shown so far in this cycle -- a handful in practice. */ -ssize_t rl_suggest_scan(t_shell *state, t_rl *l, size_t from) +** Cost: O(sg_max * line). */ +size_t rl_suggest_cands(t_shell *state, t_rl *l) { - char **a; - size_t floor; - size_t n; + char **a; + size_t i; + size_t n; + t_sg_ent e; a = (char **)state->hist.hist_cmds.ctx; n = (size_t)rl_end; - floor = 0; - if (state->hist.hist_cmds.len > (size_t)l->sg_max) - floor = state->hist.hist_cmds.len - (size_t)l->sg_max; - while (from > floor) + l->sg_scr.elem_size = sizeof(t_sg_ent); + l->sg_scr.len = 0; + i = hist_window_floor(state->hist.hist_cmds.len, l->sg_max); + while (i < state->hist.hist_cmds.len) { - from--; - if (a[from] && !ft_strncmp(a[from], rl_line_buffer, n) - && ft_strlen(a[from]) > n && sg_clean((unsigned char *)a[from]) - && !sg_offered(state, l, a[from])) - return ((ssize_t)from); + if (a[i] && !ft_strncmp(a[i], rl_line_buffer, n) + && ft_strlen(a[i]) > n && sg_clean((unsigned char *)a[i])) + { + e = (t_sg_ent){a[i], i, 0}; + vec_push(&l->sg_scr, &e); + } + i++; } - return (-1); + return (l->sg_scr.len); } -/* Forget the cycle (not the entry on screen). */ +/* Forget the cycle (not the entry on screen). The ranking buffers keep + their capacity: they are freed with the read (rl_suggest_end). */ void rl_suggest_cyc_clear(t_rl *l) { - xfree(l->sg_cyc.ctx); - l->sg_cyc.ctx = NULL; - l->sg_cyc.len = 0; - l->sg_cyc.cap = 0; + l->sg_rank.len = 0; l->sg_cyc_k = 0; xfree(l->sg_cyc_line); l->sg_cyc_line = NULL; diff --git a/src/platform/posix/rl_suggest7.c b/src/platform/posix/rl_suggest7.c index 139f4933..24fae904 100644 --- a/src/platform/posix/rl_suggest7.c +++ b/src/platform/posix/rl_suggest7.c @@ -27,41 +27,19 @@ static void sg_show(t_shell *state, size_t idx) l->sg_txt = l->sg_ent + rl_end; } -/* Start a cycle on the ghost on screen: its index is found again (it is - the newest match, see rl_suggest_find) so that only indices are kept. */ +/* Start a cycle on the ghost on screen: the ranking is made once, and + its first entry is that ghost (the history has not changed, see + rl_suggest_cyc_sync). */ static bool sg_cyc_begin(t_shell *state, t_rl *l) { - ssize_t i; - size_t idx; - - i = rl_suggest_scan(state, l, state->hist.hist_cmds.len); - if (i < 0) + if (!rl_suggest_rank(state, l)) return (false); - idx = (size_t)i; - l->sg_cyc.elem_size = sizeof(size_t); - vec_push(&l->sg_cyc, &idx); l->sg_cyc_k = 0; l->sg_cyc_line = ft_strdup(rl_line_buffer); return (true); } -/* The next older distinct match, pushed on the cycle. False at the oldest - one: there is nothing more. */ -static bool sg_push_older(t_shell *state, t_rl *l) -{ - ssize_t i; - size_t idx; - - idx = ((size_t *)l->sg_cyc.ctx)[l->sg_cyc.len - 1]; - i = rl_suggest_scan(state, l, idx); - if (i < 0) - return (false); - idx = (size_t)i; - vec_push(&l->sg_cyc, &idx); - return (true); -} - -/* Alt-N: the next older distinct match. At the oldest one there is +/* Alt-N: the next lower-ranked distinct match. At the last one there is nothing more: bell, and the ghost stays (no wrap). */ int rl_suggest_older(int count, int key) { @@ -75,15 +53,15 @@ int rl_suggest_older(int count, int key) rl_suggest_cyc_sync(state, l); if (!l->sg_cyc_line && !sg_cyc_begin(state, l)) return (rl_suggest_bell()); - if (l->sg_cyc_k + 1 >= l->sg_cyc.len && !sg_push_older(state, l)) + if (l->sg_cyc_k + 1 >= l->sg_rank.len) return (rl_suggest_bell()); l->sg_cyc_k++; - sg_show(state, ((size_t *)l->sg_cyc.ctx)[l->sg_cyc_k]); + sg_show(state, ((size_t *)l->sg_rank.ctx)[l->sg_cyc_k]); return (0); } -/* Alt-P: back to the newer alternative shown before. At the newest match - there is nothing newer: bell. */ +/* Alt-P: back to the better-ranked alternative shown before. At the best + match there is nothing better: bell. */ int rl_suggest_newer(int count, int key) { t_shell *state; @@ -97,6 +75,6 @@ int rl_suggest_newer(int count, int key) if (!l->sg_cyc_line || l->sg_cyc_k == 0) return (rl_suggest_bell()); l->sg_cyc_k--; - sg_show(state, ((size_t *)l->sg_cyc.ctx)[l->sg_cyc_k]); + sg_show(state, ((size_t *)l->sg_rank.ctx)[l->sg_cyc_k]); return (0); } diff --git a/src/platform/posix/rl_suggest8.c b/src/platform/posix/rl_suggest8.c new file mode 100644 index 00000000..a2db82b4 --- /dev/null +++ b/src/platform/posix/rl_suggest8.c @@ -0,0 +1,118 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* rl_suggest8.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "rl_private.h" +#include + +/* libc qsort, not ft_qsort: ft_qsort pivots on the last element, so a run +** of identical entries -- already in ranking order -- costs O(n^2), 6.8 s +** for one key over 20000 copies (autosuggest_test F13). libc is +** O(n log n) on glibc, musl and macOS, and both comparators here are +** total orders, so every libc gives the same ranking. */ + +/* The ranking behind the ghost and Alt-N / Alt-P: the distinct texts of +** the candidates (rl_suggest_cands), best first. No hash table: libft's +** hash dups every key, which is one malloc per candidate per key. Sorting +** the reused scratch by text puts equal texts side by side, so one walk +** scores each group; a second sort orders the groups. */ + +/* Descending order for a comparator: the larger sorts first. */ +static int sg_three(uint64_t x, uint64_t y) +{ + if (x == y) + return (0); + if (x < y) + return (1); + return (-1); +} + +/* Text first, and within one text the newest occurrence first, so a group + starts with the entry that will stand for it. */ +static int sg_cmp_text(const void *x, const void *y) +{ + const t_sg_ent *a; + const t_sg_ent *b; + int c; + + a = x; + b = y; + c = ft_strcmp(a->text, b->text); + if (c) + return (c); + return (sg_three(a->idx, b->idx)); +} + +/* Best score first; on a tie the newest. The representatives are distinct + entries, so this is a strict total order and the sort needs no + stability. */ +static int sg_cmp_rank(const void *x, const void *y) +{ + const t_sg_ent *a; + const t_sg_ent *b; + + a = x; + b = y; + if (a->score != b->score) + return (sg_three(a->score, b->score)); + return (sg_three(a->idx, b->idx)); +} + +/* Collapse each run of equal texts to its first entry (the newest) holding + the run's score, in place: the write cursor never passes the read one. + Returns the number of distinct texts. */ +static size_t sg_fold(t_sg_ent *s, size_t n, size_t len, bool recent) +{ + size_t r; + size_t w; + + r = 0; + w = 0; + while (r < n) + { + s[w] = s[r]; + s[w].score = 0; + while (r < n && !ft_strcmp(s[r].text, s[w].text)) + s[w].score = hist_score_add(s[w].score, len, s[r++].idx, + recent); + w++; + } + return (w); +} + +/* Rank the history for the line: sg_rank gets the history index of each +** distinct candidate text, best first (score, then newest). The newest +** occurrence stands for its text. Returns how many. +** +** `recent` is today's order exactly: newest first, repeats collapsed. +** Only indices are kept: the scratch holds pointers into the history +** that a later `history -c` would leave dangling. +** +** Cost: O(m log m) for m candidates; no allocation once the two vectors +** have grown to the window. */ +size_t rl_suggest_rank(t_shell *state, t_rl *l) +{ + t_sg_ent *s; + size_t n; + size_t i; + + rl_suggest_cyc_clear(l); + l->sg_rank.elem_size = sizeof(size_t); + n = rl_suggest_cands(state, l); + s = (t_sg_ent *)l->sg_scr.ctx; + qsort(s, n, sizeof(*s), sg_cmp_text); + n = sg_fold(s, n, state->hist.hist_cmds.len, l->sg_recent); + qsort(s, n, sizeof(*s), sg_cmp_rank); + i = 0; + while (i < n) + vec_push(&l->sg_rank, &s[i++].idx); + return (l->sg_rank.len); +} diff --git a/tests/autosuggest_test.py b/tests/autosuggest_test.py index 5add7bf8..1e2fa94e 100644 --- a/tests/autosuggest_test.py +++ b/tests/autosuggest_test.py @@ -16,6 +16,13 @@ matching entries, no wrap (bell at either end), reset by any edit. With no history ghost these keys are their stock readline functions. +Ranking (HELLISH_SUGGEST_STRATEGY): by default the entry with the best +frecency (how often and how recently its text was run; weight 2^16 halved +per 64 newer entries, summed per distinct text, ties to the newest) is the +ghost and the order Alt-N / Alt-P walk; `recent` is newest first. `model()` +below is the same rule written again in Python, and the cases check the +shell against it as well as against hand-computed answers. + Usage: python3 autosuggest_test.py /path/to/hellish [case-name-part [inproc]] """ import fcntl @@ -47,6 +54,7 @@ "\x06", "\x15", "\x03", "\x04", "\x12", "\x07") BKSP = "\x7f" ON = "HELLISH_AUTOSUGGEST=1" +STRAT = "HELLISH_SUGGEST_STRATEGY" try: import pyte @@ -717,17 +725,32 @@ def case_cycle(fork, tag): def case_cycle_dupes(fork, tag): - s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"]) + """The pre-frecency rule, pinned: HELLISH_SUGGEST_STRATEGY=recent.""" + s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"], + env={STRAT: "recent"}) s.keys("echo a") seen = [s.ghost()] for _ in range(4): s.keys(ALT_N) seen.append(s.ghost()) - check(tag + "30 C2 distinct entries, once each, then the end", + check(tag + "30 C2 recent: distinct entries, once each, then the end", seen == ["3", "1", "2", "2", "2"], repr(seen)) s.close() +def case_cycle_dupes_frecent(fork, tag): + """Same history, default strategy: a1 was run twice, so it leads.""" + s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"]) + s.keys("echo a") + seen = [s.ghost()] + for _ in range(4): + s.keys(ALT_N) + seen.append(s.ghost()) + check(tag + "30 C2 frecent: twice-run a1 first, once each, the end", + seen == ["1", "3", "2", "2", "2"], repr(seen)) + s.close() + + def case_cycle_reset(fork, tag): s = three(fork, ["echo ab9"]) s.keys("echo a") @@ -945,6 +968,295 @@ def case_cycle_history_shrinks(fork, tag): +def clean_text(t): + """The ghost's filter: no control byte, no DEL, no C1 (as UTF-8).""" + return not any(ord(c) < 0x20 or ord(c) == 0x7f or 0x80 <= ord(c) < 0xa0 + for c in t) + + +def model(lines, prefix, strategy="frecent", mx=1000): + """The ranking, written again: distinct texts, best first.""" + n = len(lines) + score, newest = {}, {} + for i in range(max(0, n - mx), n): + t = lines[i] + if not (t.startswith(prefix) and len(t) > len(prefix) + and clean_text(t)): + continue + if strategy == "recent": + score[t] = i + 1 + else: + score[t] = score.get(t, 0) + (65536 >> min((n - 1 - i) // 64, 16)) + newest[t] = i + return sorted(score, key=lambda t: (-score[t], -newest[t])) + + +def seeded(fork, lines, env=None, **kw): + """A shell whose history is exactly `lines` (oldest first): nothing is + run, so no other entry is in it, and autosuggest is on from the start.""" + home = tempfile.mkdtemp(prefix="autosuggest_") + with open(os.path.join(home, "hist"), "w") as f: + f.write("".join(line + "\n" for line in lines)) + e = {"HELLISH_AUTOSUGGEST": "1"} + e.update(env or {}) + return Session(home, fork=fork, env=e, **kw) + + +def walk(s, typed, steps): + """The ghost after `typed`, then after each of `steps` Alt-N.""" + s.keys(typed) + seen = [s.ghost()] + for _ in range(steps): + s.keys(ALT_N) + seen.append(s.ghost()) + return seen + + +# re is run three times, clean twice and newest, test once: frecency says +# re, the newest entry says clean. +ISSUE = ["make re", "make test", "make re", "make clean", "make re", + "make clean"] + + +def case_frecent_issue(fork, tag): + s = seeded(fork, ISSUE) + got = walk(s, "make ", 3) + check(tag + "37 F1 frecent: the most-run command, not the newest", + got == ["re", "clean", "test", "test"], repr(got)) + s.close() + s = seeded(fork, ISSUE, env={STRAT: "recent"}) + got = walk(s, "make ", 3) + check(tag + "37 F1 recent: the newest, repeats collapsed", + got == ["clean", "re", "test", "test"], repr(got)) + s.close() + + +def case_frecent_recent(fork, tag): + lines = ["echo a1", "echo b1", "echo a2", "echo b2", "echo a3"] + s = seeded(fork, lines, env={STRAT: "recent"}) + got = walk(s, "echo ", 4) + check(tag + "38 F2 recent is newest first, as before frecency", + got == ["a3", "b2", "a2", "b1", "a1"], repr(got)) + s.close() + + +def case_frecent_default(fork, tag): + for val in ("", "bogus", "RECENT", "recent "): + s = seeded(fork, ISSUE, env={STRAT: val}) + got = walk(s, "make ", 0) + check(tag + "39 F3 %r is frecent" % val, got == ["re"], repr(got)) + s.close() + s = seeded(fork, ISSUE) + got = walk(s, "make ", 0) + check(tag + "39 F3 unset is frecent", got == ["re"], repr(got)) + s.close() + + +def case_frecent_live(fork, tag): + s = seeded(fork, ISSUE) + steps = (("", "re"), (STRAT + "=recent", "clean"), + (STRAT + "=bogus", "re"), (STRAT + "=recent", "clean"), + ("unset " + STRAT, "re")) + for cmd, want in steps: + if cmd: + s.run(cmd) + got = walk(s, "make ", 0) + check(tag + "40 F4 after %r the ghost is %r" % (cmd, want), + got == [want], repr(got)) + s.keys(CTRL_U) + s.close() + + +def case_frecent_tie(fork, tag): + lines = ["echo x2", "echo x1", "echo x2", "echo x1"] + s = seeded(fork, lines) + got = walk(s, "echo x", 2) + check(tag + "41 F5 equal scores: the newest first", + got == ["1", "2", "2"], repr(got)) + s.close() + s = seeded(fork, lines[1:] + ["echo x2"]) + got = walk(s, "echo x", 2) + check(tag + "41 F5 equal scores, other way round", + got == ["2", "1", "1"], repr(got)) + s.close() + + +def decay_lines(total): + """fa twice at the very start, filler, fb newest: `total` entries.""" + lines = ["echo fa", "true 0", "echo fa"] + lines += ["true %d" % n for n in range(1, total - 3)] + return lines + ["echo fb"] + + +def case_frecent_decay(fork, tag): + # Two fa at ages total-1 and total-3. Weight 2^16 halves each 64 newer + # entries: at 66 entries (ages 65, 63) 2^15 + 2^16 beats fb's 2^16; at + # 67 (ages 66, 64) both have halved, 2^16 ties, and the newest wins. + for total, want in ((20, "fa"), (66, "fa"), (67, "fb"), (300, "fb")): + lines = decay_lines(total) + s = seeded(fork, lines) + got = walk(s, "echo f", 0) + check(tag + "42 F6 %d entries: %s wins" % (total, want), + got == [want[1]] and model(lines, "echo f")[0] == "echo " + want, + repr(got)) + s.close() + + +def case_frecent_window(fork, tag): + lines = ["echo wa", "echo wb", "echo wa", "echo wb", "echo wa", + "echo wc", "echo wb"] + s = seeded(fork, lines) + got = walk(s, "echo w", 0) + check(tag + "43 F7 whole history: wa 3, wb 3, wc 1: the tie goes to " + "the newest", got == ["b"], repr(got)) + s.close() + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": "3"}) + got = walk(s, "echo w", 3) + check(tag + "43 F7 MAX=3: wa, wc, wb once each, newest first", + got == ["b", "c", "a", "a"], repr(got)) + s.close() + lines = ["echo wa", "echo wa", "echo wb", "echo wa", "echo wb", + "echo wc"] + s = seeded(fork, lines) + got = walk(s, "echo w", 0) + check(tag + "43 F7 MAX unset: wa x3 wins", got == ["a"], repr(got)) + s.close() + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": "4"}) + got = walk(s, "echo w", 2) + check(tag + "43 F7 MAX=4: the two older wa do not count, wb x2 wins", + got == ["b", "c", "a"], repr(got)) + s.close() + + +def case_frecent_cycle(fork, tag): + s = seeded(fork, ISSUE) + s.keys("make ") + order = [] + for _ in range(3): + order.append(s.ghost()) + s.keys(ALT_N) + check(tag + "44 F8 Alt-N follows the score", order == ["re", "clean", + "test"] and s.ghost() == "test", repr(order)) + s.keys(ALT_N) + check(tag + "44 F8 bell at the last, ghost stays", + b"\a" in s.since() and s.ghost() == "test", repr(s.since())) + s.keys(ALT_P + ALT_P) + check(tag + "44 F8 Alt-P walks back to the best", + s.ghost() == "re" and s.row() == PROMPT + "make re", + repr(s.row())) + s.keys(ALT_P) + check(tag + "44 F8 bell at the best, no wrap", + b"\a" in s.since() and s.ghost() == "re", repr(s.since())) + s.close() + + +def case_frecent_dirty(fork, tag): + lines = ["echo ok", "echo \x1b[31mred", "echo \x1b[31mred", + "echo \x1b[31mred", "echo ok2"] + s = seeded(fork, lines) + got = walk(s, "echo ", 3) + check(tag + "45 F9 a frequent entry holding ESC is never offered", + got == ["ok2", "ok", "ok", "ok"] and b"\x1b[31m" not in s.buf, + repr(got)) + s.close() + + +def case_frecent_history_shrinks(fork, tag): + s = seeded(fork, ISSUE) + s.run("bind -x '\"\\C-t\": history -c'") + s.keys("make ") + s.keys(ALT_N) + check(tag + "46 F10 cycled before the clear", s.ghost() == "clean", + repr(s.ghost())) + s.keys("\x14") + s.keys(ALT_N) + check(tag + "46 F10 no ghost survives a cleared history (Alt-N)", + s.ghost() == "" and s.alive(), repr(s.ghost())) + s.keys(CTRL_U + "make ") + s.keys(ALT_N + ALT_P) + check(tag + "46 F10 nor a cycle begun afterwards", + s.ghost() == "" and s.alive(), repr(s.ghost())) + s.keys("\x07\x15") + s.run("echo ok") + check(tag + "46 F10 the shell still runs lines", + "ok" in s.screen_text(), repr(s.screen_text()[-200:])) + s.close() + + +def case_frecent_typing(fork, tag): + long_a = "echo abcdefghijklmnopqrstuvwxyz0123456789" + long_b = "echo abcdefgXYZ0123456789abcdefghij" + lines = [long_a, long_b, long_a, "echo abc", long_a] + s = seeded(fork, lines) + bad = [] + for path in (long_a, long_b): + s.keys(CTRL_U + path[:4]) + for k in range(5, len(path) + 1): + s.keys(path[k - 1]) + want = model(lines, path[:k]) + want = want[0][k:] if want else "" + if s.ghost() != want: + bad.append((path[:k], s.ghost(), want)) + check(tag + "47 F11 typing forward: every ghost is the best match for " + "what is typed (%d keys)" % (len(long_a) + len(long_b) - 10), + not bad, repr(bad[:3])) + s.close() + + +def random_history(rng): + words = ["alpha", "alps", "alt", "amber", "ant"] + n = rng.choice([8, 40, 70, 130, 200]) + lines = [] + for i in range(n): + w = rng.choice(words) + if rng.random() < 0.5: + lines.append("echo %s" % w) + else: + lines.append("echo %s %d" % (w, rng.randrange(3))) + if i and lines[-1] == lines[-2]: + lines[-1] += " x" + return lines + + +def case_frecent_model(fork, tag): + import random + rng = random.Random(140 + int(fork)) + for k in range(5): + lines = random_history(rng) + mx = rng.choice([1000, 1000, 50]) + prefix = rng.choice(["echo ", "echo al", "echo a"]) + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": str(mx)}) + want = model(lines, prefix, mx=mx) + got = walk(s, prefix, len(want) + 1) + exp = [t[len(prefix):] for t in want] + exp = (exp + exp[-1:] * 2)[:len(want) + 2] if exp else [""] + exp = exp[:len(got)] if exp != [""] else [""] * len(got) + check(tag + "48 F12 random history %d (%d entries, max %d, %r): " + "the cycle equals the model" % (k, len(lines), mx, prefix), + got == exp, "got %r\nwant %r" % (got, exp)) + s.close() + + + +def case_frecent_sorted_worst(fork, tag): + """20000 copies of one entry arrive already in ranking order, the worst + case of a last-element-pivot quicksort (O(n^2)): the first key must + still paint its ghost quickly.""" + n = 20000 + s = seeded(fork, ["ls -la"] * n + ["pwd"], + env={"HELLISH_SUGGEST_MAX": str(n + 1), + "HISTSIZE": str(n + 10), "HISTFILESIZE": str(n + 10)}) + s.mark = len(s.buf) + t0 = time.time() + os.write(s.fd, b"l") + s.until(lambda: s.ghost() == "s -la", 20.0) + dt = time.time() - t0 + check(tag + "37 F13 sorted worst case paints within 1.5 s", + s.ghost() == "s -la" and dt < 1.5, "%r %.2fs" % (s.ghost(), dt)) + s.close() + + + CASES = (case_off_by_default, case_on, case_accept, case_enter_plain, case_newest, case_edit, case_cursor, case_interrupt, case_history_keys, case_search, case_vi, case_escape, @@ -952,10 +1264,16 @@ def case_cycle_history_shrinks(fork, tag): case_paste, case_max, case_unset, case_user_bind, case_postdisplay, case_resize, case_dumb, case_word, case_word_ctrl_right, case_word_blanks, case_word_stock, - case_word_utf8, case_cycle, case_cycle_dupes, case_cycle_reset, + case_word_utf8, case_cycle, case_cycle_dupes, + case_cycle_dupes_frecent, case_cycle_reset, case_cycle_accept, case_cycle_max, case_cycle_stock, case_cycle_post, case_cycle_user_bind, case_cycle_off, - case_cycle_vi, case_cycle_history_shrinks) + case_cycle_vi, case_cycle_history_shrinks, case_frecent_issue, + case_frecent_recent, case_frecent_default, case_frecent_live, + case_frecent_tie, case_frecent_decay, case_frecent_window, + case_frecent_cycle, case_frecent_dirty, + case_frecent_history_shrinks, case_frecent_typing, + case_frecent_model, case_frecent_sorted_worst) def main(): diff --git a/tests/complete_make_test.py b/tests/complete_make_test.py index 236da2a9..53fab79b 100755 --- a/tests/complete_make_test.py +++ b/tests/complete_make_test.py @@ -96,7 +96,7 @@ def drain_idle(fd, chunks, quiet=0.35, cap=8.0): return chunks -def spawn(cwd, home): +def spawn(cwd, home, cols=200, extra=None): env = { "HOME": home, "PATH": "/usr/bin:/bin", "TERM": "xterm-256color", "LANG": "C.UTF-8", "PS1": "$ ", @@ -106,6 +106,7 @@ def spawn(cwd, home): "ASAN_OPTIONS": "detect_leaks=%d" % (1 if ASAN else 0), } env.update(EXTRA) + env.update(extra or {}) pid, fd = pty.fork() if pid == 0: # nothing may leave this block except _exit (see @@ -118,16 +119,22 @@ def spawn(cwd, home): except BaseException: pass os._exit(127) - fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", 60, 200, 0, 0)) + fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", 60, cols, 0, 0)) return pid, fd -def session(cwd, steps, settle=0.6): +def session(cwd, steps, settle=0.6, hist=None, env=None, cols=200): """Run `steps` (bytes to type, or a callable run between keystrokes) - in a fresh shell; return the output of each bytes step, in order.""" + in a fresh shell; return the output of each bytes step, in order. + `hist` is the history the shell starts with, oldest first.""" home = tempfile.mkdtemp(prefix="hellish_mk_home_") TMP.append(home) - pid, fd = spawn(cwd, home) + extra = dict(env or {}) + if hist is not None: + extra["HISTFILE"] = os.path.join(home, "hist") + with open(extra["HISTFILE"], "w") as f: + f.write("".join(h + "\n" for h in hist)) + pid, fd = spawn(cwd, home, cols, extra) chunks = [] drain_idle(fd, chunks, quiet=0.4, cap=6.0) outs = [] @@ -177,6 +184,19 @@ def listing(out): return set(ANSI.sub("", block).split()) +def ordered(out): + """Like listing(), but in the order printed. Needs a terminal so + narrow (session(..., cols=10)) that readline prints one name a row.""" + i = out.find("\a\r\n") + if i < 0: + return [] + block = out[i + 3:] + j = block.find("\r\n$ ") + if j >= 0: + block = block[:j] + return ANSI.sub("", block).split() + + def line_is(out, want): return want in ANSI.sub("", out).replace("\a", "") @@ -453,6 +473,86 @@ def noperm(d): b.stdout, b.returncode)) +# `re` is run three times, `clean` twice and last, `test` once: frecency +# says re, clean, test; the newest entry says clean, re, test. +HIST = ["make re", "make test", "make re", "make clean", "make re", + "make clean"] +RANKED = "all clean fclean lint re test".split() +UNUSED = ["all", "fclean", "lint"] + + +def ranked_listing(hist, typed=b"make ", env=None, files=None): + d = fixture(dict({mk: "".join("%s:\n" % t for t in RANKED)}, + **(files or {}))) + out = session(d, [typed + b"\t\t\t"], hist=hist, env=env, cols=10)[0] + return ordered(out) + + +def ranked_cases(): + # 25 + got = ranked_listing(HIST) + check("25 used targets by frecency, then the rest alphabetically", + got == ["re", "clean", "test"] + UNUSED, str(got)) + got = ranked_listing(HIST, env={"HELLISH_SUGGEST_STRATEGY": "recent"}) + check("25 STRATEGY=recent: used targets newest first", + got == ["clean", "re", "test"] + UNUSED, str(got)) + got = ranked_listing([]) + check("25 no history: alphabetical", got == RANKED, str(got)) + got = ranked_listing(["echo re", "ls test", "cmake lint", "make"]) + check("25 only invocations of make count", got == RANKED, str(got)) + got = ranked_listing(["gmake lint", "/usr/bin/make test", "make lint"]) + check("25 gmake and a path to make count", + got == ["lint", "test", "all", "clean", "fclean", "re"], str(got)) + got = ranked_listing(["make re re re", "make lint test", "make test"]) + check("25 a word repeated inside one entry counts once", + got == ["test", "lint", "re", "all", "clean", "fclean"], str(got)) + out = comp_hist(HIST, b"make cl") + check("25 make cl still completes", line_is(out, "make clean "), + repr(out[-80:])) + + # 26 + got = ranked_listing(["make re", "make re", "make re", "make fclean", + "make test"], env={"HELLISH_SUGGEST_MAX": "2"}) + check("26 the window bound applies to make too (MAX=2)", + got == ["test", "fclean", "all", "clean", "lint", "re"], str(got)) + + # 27 + many = {"f%02d" % n: "x" for n in range(30)} + d = fixture(dict({mk: "".join("%s:\n" % t for t in RANKED)}, **many)) + outs = session(d, [b"make \t\t\t", KILL, b"ls f\t\t\t", KILL, + b"bas\t\t\t"], hist=HIST, cols=10) + mk_order = ordered(outs[0]) + files = ordered(outs[2]) + cmds = ordered(outs[4]) + check("27 make TAB ranked", mk_order == ["re", "clean", "test"] + + UNUSED, str(mk_order)) + check("27 then a filename TAB: sorted again, not left unsorted", + len(files) >= 30 and files == sorted(files), str(files[:8])) + check("27 then a command TAB: sorted again", + len(cmds) >= 2 and cmds == sorted(cmds), str(cmds[:8])) + + # 28 + d = fixture({mk: "all:\nclean:\n"}) + outs = session(d, [b"shopt -s progcomp\n", + b"complete -W 'zz yy' make\n", b"make \t\t\t"], + hist=["make clean", "make all", "make clean"]) + check("28 a user complete -W still wins over the ranking", + listing(outs[2]) == {"zz", "yy"}, str(sorted(listing(outs[2])))) + + # 29 + files = {"sub/Makefile": "s1:\n", "other.mk": "o1:\n"} + for typed in (b"make -C sub ", b"make -f other.mk "): + got = ranked_listing(HIST, typed, files=files) + check("29 %s keeps filenames with a history" % typed.decode(), + set(got) == {mk, "sub/", "other.mk"} + and got == sorted(got), str(got)) + + +def comp_hist(hist, typed): + d = fixture({mk: "".join("%s:\n" % t for t in RANKED)}) + return session(d, [typed + b"\t"], hist=hist)[0] + + def main(): if not os.access(SHELL, os.X_OK): sys.exit("complete_make_test: not executable: %s" % SHELL) @@ -465,6 +565,8 @@ def main(): EXTRA.clear() EXTRA.update(env) cases() + if not ASAN: + ranked_cases() finally: for d in TMP: shutil.rmtree(d, ignore_errors=True) diff --git a/tests/frontend_budget_test.py b/tests/frontend_budget_test.py index 39918ef5..f0563182 100755 --- a/tests/frontend_budget_test.py +++ b/tests/frontend_budget_test.py @@ -46,6 +46,9 @@ MARK = b"#> " ENTERS = 20 KEYS = 40 +ROUNDS = 12 +DIM = b"\x1b[2m" +HIST_N = 1000 TRACE_LINE = re.compile(r"^(\d+)\s+(\d+\.\d+)\s+([a-z_0-9]+)") # name -> (per Enter, per key, idle total, writes per resize phase) @@ -63,6 +66,11 @@ BUDGET["in-process/as"] = dict(BUDGET["in-process"], nomatch_write=1.2) BUDGET["forked/as+rp"] = dict(BUDGET["forked"], nomatch_write=2.2) BUDGET["forked/as"] = dict(BUDGET["forked"], nomatch_write=1.2) +# Frecency (#140): a history of 1000 distinct entries, all matching the +# first key, so every Backspace ranks them all. Same per-key budget: the +# ranking is CPU, and reuses its buffers instead of asking the kernel. +BUDGET["in-process/frecent"] = dict(BUDGET["in-process"], nomatch_write=1.2) +BUDGET["forked/frecent"] = dict(BUDGET["forked"], nomatch_write=1.2) SEED = "x" * 60 AS_ON = {"HELLISH_AUTOSUGGEST": "1"} FORKING = ("clone", "clone3", "fork", "vfork", "pipe2", "pipe", "wait4", @@ -77,7 +85,7 @@ def check(name, ok, detail=""): class Traced: - def __init__(self, home, trace, extra, seed=()): + def __init__(self, home, trace, extra, seed=(), hist=()): env = { "HOME": home, "PATH": os.environ.get("PATH", "/usr/bin:/bin"), "TERM": "xterm-256color", "LANG": "C.UTF-8", "LC_ALL": "C.UTF-8", @@ -87,6 +95,10 @@ def __init__(self, home, trace, extra, seed=()): "HISTFILE": "/dev/null", "ASAN_OPTIONS": "detect_leaks=0", } env.update(extra) + if hist: + env["HISTFILE"] = os.path.join(home, "hist") + with open(env["HISTFILE"], "w") as f: + f.write("".join(h + "\n" for h in hist)) self.buf = bytearray() self.phases = [] self.pid, self.fd = pty.fork() @@ -148,6 +160,19 @@ def typing(self, ch=b"x"): os.write(self.fd, b"\x15") self.settle() + def scanning(self): + """Forward keys and Backspace over a long history: each Backspace + ends a cached answer, so the ranking runs; forward keys must not.""" + self.mark = len(self.buf) + for _ in range(ROUNDS): + for key in (b"x", b"x", b"x", b"x", b"\x7f"): + os.write(self.fd, key) + self.pump(0.03) + self.settle() + self.dim = DIM in bytes(self.buf[self.mark:]) + os.write(self.fd, b"\x15") + self.settle() + def idle(self): time.sleep(2.0) @@ -184,14 +209,16 @@ def counts(lines, t0, t1): return c -def cell(label, extra, base, seed=()): +def cell(label, extra, base, seed=(), hist=()): trace = os.path.join(base, "trace-" + label.replace("/", "_")) home = tempfile.mkdtemp(dir=base) - s = Traced(home, trace, extra, seed) + s = Traced(home, trace, extra, seed, hist) s.phase("enter", s.enters) s.phase("idle", s.idle) s.phase("key", s.typing) s.phase("nomatch", lambda: s.typing(b"q")) + if hist: + s.phase("scan", s.scanning) s.phase("winch", s.winch_same) s.phase("rows", s.rows_only) s.close() @@ -234,6 +261,19 @@ def cell(label, extra, base, seed=()): check("%s: a key with nothing to paint does at most %.1f writes" % (label, b["nomatch_write"]), n <= b["nomatch_write"], "%.1f per key" % n) + if hist: + n = sum(by["scan"].values()) / float(ROUNDS * 5) + w = by["scan"]["write"] / float(ROUNDS * 5) + print("# %s: %d distinct entries, per key %.1f syscalls, " + "%.1f writes" % (label, len(hist), n, w)) + check("%s: a key over %d distinct entries, Backspace included, " + "costs at most %d syscalls" % (label, len(hist), b["key"]), + n <= b["key"], "%.1f per key: %s" % ( + n, dict(by["scan"].most_common(6)))) + check("%s: and at most %d writes" % (label, b["key_write"]), + w <= b["key_write"], "%.1f per key" % w) + check("%s: the ghost was painted over that history" % label, + s.dim, "no dim ghost in the output") check("%s: an idle prompt costs nothing" % label, sum(by["idle"].values()) == 0, "%d syscalls while idle: %s" % (sum(by["idle"].values()), @@ -258,6 +298,10 @@ def main(): cell("forked/as+rp", dict(AS_ON, HELLISH_RL_FORK="1"), base, seed) cell("forked/as", dict(AS_ON, RPROMPT="", HELLISH_RL_FORK="1"), base, seed) + hist = ["xxxx%04d" % n for n in range(HIST_N)] + cell("in-process/frecent", dict(AS_ON, RPROMPT=""), base, (), hist) + cell("forked/frecent", dict(AS_ON, RPROMPT="", HELLISH_RL_FORK="1"), + base, (), hist) finally: shutil.rmtree(base, ignore_errors=True) if FAILS: diff --git a/wiki/manual.md b/wiki/manual.md index 6a6d6592..2fb57448 100644 --- a/wiki/manual.md +++ b/wiki/manual.md @@ -175,6 +175,7 @@ inside the shell: `help` for the grouped list, `help NAME` for one, | `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down walk plain history; default on = search by the text before the cursor; read at every prompt | | `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion (off by default): the newest history line starting with the typed text, dim after the cursor; Right / End / Ctrl-F insert it, Alt-F / Ctrl-Right insert its next word (leading blanks + the run up to the next space or tab), Alt-N / Alt-P cycle through older / newer distinct matches (bell at either end, no wrap; any edit resets); never a line with control characters or a newline; read at every prompt | | `HELLISH_SUGGEST_MAX=N` | history lines the suggestion scans, newest first (default 1000; non-numeric or below 1 = default); read at every prompt | +| `HELLISH_SUGGEST_STRATEGY=recent` | newest-first suggestions; default (unset/other) = frecency (run count weighted toward recent runs), also ordering `make `; read at every prompt | | `POSTDISPLAY` | zle widgets only: the text painted after the cursor, i.e. the suggestion; assign it to show your own until the buffer next changes | | `HELLISH_NO_UPDATE_CHECK=1` | never check for releases; no badge | | `HELLISH_NO_ANIM=1` | skip the startup animation |