AGENDA

1. 01Progress update - Olivier's email
2. 02Recent news - Jev, System One Models
3. 03My findings
4. 04Current limitations
5. 05Next steps
6. 06Action items & next meeting

OLIVIER'S EMAIL - WHERE THINGS STAND

Infrastructure is on hold for now. My effort is focused on Point 1 of your email: designing and building a per-task-type cost model.

• 01
  PER-TASK-TYPE COST MODELDESIGNING + BUILDING IT NOW
  ACTIVE
• 02
  JAYSON'S INFRASTRUCTURE
  ON HOLD
• 03
  DEFINITION OF FAIRNESS
  ON HOLD
• 04
  KEDA SCALING POLICIES
  ON HOLD
• 05
  TASK SCHEMA (ATTRIBUTES)ALONGSIDE POINT 1
  ACTIVE
• 06
  HISTORICAL DATA (DATASET)ALONGSIDE POINT 1
  ACTIVE

TYPESAFE'S JEV - SYSTEM ONE MODELS

• TRADITIONAL CLASSIFIERSFast, but not smart enough.
  FASTNOT SMART ENOUGH
• AUTOREGRESSIVE MODELSSmart, but not fast. BOUNDARYML.COM
  SMARTNOT FAST
• SYSTEM ONE MODELSA single forward pass (non-autoregressive) with access to the embedding space of a pretrained LLM - Qwen, GPT, Kimi, etc.
  FAST: SINGLE PASSSMART: PRETRAINED EMBEDDINGS

SYSTEM ONE INTERFACES

Every interface returns a probability or a distribution over an answer space you supply.

EVALUATION SETUP - 5896 COMBINATIONS

• OPEN-JEV-2B~600 MS INFERENCE
• OPEN-JEV-DEBERTA-LARGE-V3~40 MS INFERENCE
• LAYA-421M~25 MS INFERENCE

WHY SYSTEM ONE MODELS FOR COST?

The purpose of the experiment: check whether these models provide a meaningful heuristic as a per-task-type cost model.

1. 01
   They estimate cost. A Score over GB-s levels yields a cost estimate for every file/task combination.
2. 02
   They flex across criteria. A variable number of criteria is handled through the Score or Choice interface.

KEY FINDINGS - THE SIZE AXIS

Open-Jev-2B is the only model that orders expected cost by file size - System One Models feel like a favorable direction.

CURRENT LIMITATIONS

1. 01
   Finetuned for human-facing tasks. Banking, sentence sentiment, invoice processing, security incidents, customer service, agent-trace observability.NONE OF THOSE LOOK LIKE ASYNCHRONOUS SCHEDULING
2. 02
   Even the #1 trending model falls flat. Open-Jev-DeBERTa-large-v3 is #1 trending on HuggingFace as of mid September - but from an initial glance, its outputs do not look useful as a computational-cost heuristic.MEAN E IS FLAT ACROSS ALL 8 FILE SIZES (2.34 - 2.59)
3. 03
   The architecture is less than a month old. There has yet to be an application to asynchronous scheduling tasks - let alone as a heuristic to estimate computational cost.
4. 04
   The promising model doubts itself. 47% of Open-Jev-2B predictions report 0.000 confidence.

NEXT STEPS

1. 01
   Build a novel dataset - file vs computational cost.FILE · METADATA · GB-S · DURATION · MEMORY USAGE · CPU/GPU USAGE
2. 02
   Evaluate larger + other System One Models on school compute.

   OPEN-JEV-9BBESPOKE-NIMBLE-9B-V2KEV-9BMEANBLOCK/JEV-CPU (CPU ONLY)

   WATCH FOR MODEL DRIFT - FILETYPES CHANGE
3. 03
   Evaluate Natural Language Inference classifiers as an alternative heuristic family.
4. 04
   Vary prompting techniques + feature engineering.I WOULD LOVE TO INJECT FILE ENTROPY AS A FEATURE
5. 05
   Search for existing datasets + benchmarks before building our own from scratch.
6. 06
   Put backlog + normalized wait time in as attributes/criteria/features (collection length included).
7. 07
   Further statistical analysis on the existing evaluation.

EXTRA ITEMS - DR. BAYSAL

1. 01
   IDP email received. I saw your Individual Development Plan email - just letting you know it arrived.
2. 02
   GSPMD finished. I finished reading “GSPMD: General and Scalable Parallelization for ML Computation Graphs.”IF SUMMER-SCHEDULE MEETINGS CONTINUE, I WOULD LOVE TO DISCUSS MORE LITERATURE

ACTION ITEMS

1. 01
   Any of the next steps. Whichever items you think are most valuable.
2. 02
   Continue the literature review.STILL ADJUSTING THE LIT REVIEW LIST

FEEDBACK - IFS

Feedback given by IFS on this work - the answer to the ask.

1. 01
   If the state space is sparse/small, memoize it. The number of possible states (filetype × filesize) can be calculated in full (grid search), then a lookup table finds its cost/score.EXPENSIVE INITIALLY - SAVES FUTURE COSTS (MEMOIZATION)
   JUSTIN
2. 02
   Place more emphasis on XML file types. Do more research in this area specifically.
3. 03
   Test against autoregressive models too. Instead of just System One models, see if performance improves.INTUITIVELY, IT SEEMS LIKE LATENCY WILL INCREASE
4. 04
   Integrate this with the mock infrastructure.
   ASAP
5. 05
   Search before building. Before building a novel dataset (Step 1), perform Step 5 - search for existing datasets + benchmarks.
6. 06
   Explore LLMs + evaluate their performance on this task.ALSO PREFERS WE DON'T GO IN THE DIRECTION OF STEP 3 (EVALUATE NLI CLASSIFIERS)
   DR. BAYSAL
7. 07
   Finetuning System One models. There is promise in this direction.
   DR. BAYSAL
8. 08
   Apply current models to the existing mock infrastructure.
   HIGH PRIORITY

NEXT MEETING