Rubric Categories and Criteria

1. Authenticity and Specificity (Weight: 30 points)

Focus: Detects real customer stories vs. generic/AI-like text. Looks for unique details, personal anecdotes, and avoidance of repetitive templates.

Semantic/Entity Angle: Scan for named entities (e.g., specific dates, weather events, roofing services like roof replacement, roof repair, roof installation, or roofing materials) using entity recognition. Generic reviews lack these.

Scoring Breakdown:

20-30: High specificity with personal story (e.g., “They replaced my leaking asphalt shingles after the big storm in Syracuse last June—saved my basement from flooding!”). Includes local vernacular (e.g., “y’all,” to,” or region-specific issues like “NY heat warping the tiles”).

10-19: Moderate details (e.g., mentions a problem/solution but no unique context, like “Fixed my roof leak quickly”).

0-9: Generic or suspicious (e.g., “Great service! Highly recommend.”—certain keywords or patterns means it could be AI; lacks entities or depth).

Detection Tips: Use semantic tools to check for entity density (aim for 2+ per review). Flag patterns like identical phrasing across reviews as fake.

2. Semantic Relevance and Themes (Weight: 25 points)

Focus: Evaluates how well the content ties to roofing topics, using semantic analysis for key themes (e.g., durability, customer service).

Semantic/Entity Angle: Identify roofing entities (e.g., “metal roofing,” “gutter installation”) and semantic clusters (e.g., positive sentiment around “timeliness” or “quality materials”). This boosts entity SEO by highlighting authoritative terms.

Scoring Breakdown:

15-25: Strong thematic depth with relevant entities (e.g., “Their crew used CertainTeed shingles that held up against our humid New York summers—no more mold issues!”). Themes like “weather resistance” or “emergency repairs” recur authentically.

8-14: Basic relevance (e.g., mentions “roof repair” but shallow, like “Did a good job on the roof”).

0-7: Off-topic or vague (e.g., no roofing specifics; just “amazing company”).

Detection Tips: Break into semantic elements—positive/negative sentiment via tools (e.g., VADER for polarity). Count roofing-related entities (target 1-3 per review for authenticity).

3. Emotional and Narrative Depth (Weight: 20 points)

Focus: Values emotional language and storytelling, which fake reviews often lack (AI tends to be flat).

Semantic/Entity Angle: Look for sentiment-laden entities (e.g., “frustrating leak” turning to “relieved after fix”). Semantic blah blah: Analyze for narrative arcs (problem → solution → outcome).

Scoring Breakdown:

14-20: Rich narrative with emotion (e.g., “I was stressed about the hail damage, but these guys showed up same-day and made it like new—total lifesavers!”).

7-13: Some emotion/story (e.g., “Happy with the results after a tough job”).

0-6: Neutral or robotic (e.g., “Service was satisfactory”—lacks feeling or flow).

Detection Tips: Semantic sentiment analysis—score for polarity (positive >0.5) and subjectivity (high for personal stories). Local vernacular adds bonus points for cultural fit.

4. Local and Contextual Fit (Weight: 15 points)

Focus: Prioritizes reviews with city/state-specific references, enhancing geo-targeted SEO.

Semantic/Entity Angle: Entity recognition for places (e.g., “Hurricane Sally damage”). Semantic: Check if context matches the city’s climate/roofing needs.

Scoring Breakdown:

10-15: Strong local ties (e.g., “Perfect for our Central New York storms; they know how to handle pine tree debris on roofs”).

5-9: Mild references (e.g., mentions the state but not specifics).

0-4: No location context (e.g., could apply anywhere—suspect for fakes).

Detection Tips: Use named entity recognition (NER) to tag GPE (geo-political entities). Cross-check with city data (e.g., common weather issues).

5. Overall Credibility and Balance (Weight: 10 points)

Focus: Assesses if the review feels balanced (mentions pros/cons) and responds well to company replies.

Semantic/Entity Angle: Semantic variety—mixed sentiment avoids “too perfect” fakes. Entities should align with known facts (e.g., company services).

Scoring Breakdown:

7-10: Balanced and credible (e.g., “A bit pricey, but worth it for the warranty on those Owens Corning materials”).

4-6: Mostly positive but believable.

0-3: Overly effusive or inconsistent (e.g., contradicts other reviews semantically).

Detection Tips: Compare semantic similarity across a company’s reviews—high duplication flags fakes.

Review Content Rubric (v2)

Total: 100 points

How to Use This Rubric

Read the review once → Does it have a clear Problem → Solution → Result story?

1. → If no → auto-cap at 45 total points (still usable but won’t carry the ranking).
2. Scoring Sheet Example (you can make this in Google Sheets)
   • Column A: Review Text
   • Column B: Problem–Solution Score (0-30)
   • Column C: Local Vernacular Score (0-25)
   • Final Column = SUM

Entity + Semantic Boost (for SEO value)

1. While scoring, also tag entities that will later appear in your list copy:
   • Brands: GAF, CertainTeed, Owens Corning
   • Materials: Timberline HDZ, ice-and-water shield, drip edge
   • Local triggers: “Syracuse weather”, “San Jose humidity”, “Wiregrass summers”, “red clay dirt under the house” “yall” etc