timmy opened issue timmy/harness#3

timmy
Thu, 02 Apr 2026 18:57:45 +0000

Test Gemma 4 31B - Long Context

This issue tracks testing of 8K context window handling using the Gemma 4 31B model via the Hermes Profile.

Profile Location

~/.hermes/profiles/gemma4/

Test Script

#!/bin/bash
# test_gemma4_31b_context.sh
# Long context test for Gemma 4 31B model

set -e

PROFILE_PATH="$HOME/.hermes/profiles/gemma4"
MODEL_NAME="gemma-4-31b-it"

echo "Testing Gemma 4 31B Long Context (8K)..."

# Load profile
source "$PROFILE_PATH/env"

# Generate a long context (~6K tokens)
LONG_TEXT=$(python3 -c "print(\"This is a test sentence. \"*500)")

# Test long context comprehension
hermes run --model "$MODEL_NAME" --max-tokens 8192 --prompt "Summarize the following text in 3 bullet points:

$LONG_TEXT"

# Test context window edge cases
hermes run --model "$MODEL_NAME" --prompt "Given this story:
$LONG_TEXT

What is the main theme?"

echo "Test completed successfully!"

Test Checklist

Profile loads correctly

Model handles 8K context without truncation

Long document summarization works

Context retention across long prompts

Performance remains acceptable with large context

Assignee

@ezra