Berkeley Civic Data Class
Instructional Overview

Berkeley Housing Pipeline – Annual Progress Notebooks  

January 4, 2026  

This project is building a growing series of Jupyter Notebooks designed to generate and enhance the City of Berkeley’s Annual Housing Element Progress Report, with new visual elements and additional Datasette-based analytic modules. The focus is on measuring actual permitting, construction, economic, and neighborhood outcomes of recent housing policies.

The workflow is organized as a modular notebook collection rather than a single master file. Instead of one monolithic notebook, the master analysis is being refactored into independent notebooks for each stage of the pipeline: data acquisition and cleaning; statistical analysis; visualization and mapping; and final report generation. Each notebook is intended to be readable, remixable, and suitable for instruction, collaboration, and public review.

These computational notebooks are meant to complement and extend the City of Berkeley’s official Annual Housing Element Progress Report to California HCD, not replace it. The repository is built as a reusable template that can be cloned from GitHub, then fully customized and run locally or in hosted environments such as Google Colab or Binder. All SQL-ready data outputs are exposed through Datasette, enabling exploratory analysis, rapid prototyping of visualizations, and integration into other tools.

For Berkeley High and UC Berkeley classes, as well as local media and data journalism outlets such as Berkeleyside,  The Berkeley Scanner, the Daily Cal, and similar organizations, the Datasette instances and underlying SQLite databases provide a shared analytic backbone. Large language models can be used to generate or refine SQL queries that run directly against these datasets in Datasette, making it easier to build interactive maps, neighborhood profiles, time-series charts of permitting and construction, and targeted statistical analyses without deep database expertise.

As the City of Berkeley expands and stabilizes its API access to OpenGov and related systems covering Planning, Building, Design Review, and Certificates of Occupancy, the notebooks and Datasette instances will incorporate richer, more timely data. The roadmap includes integrating Berkeley business licenses, public energy and water use data, and modules focused on displacement risk, affordability outcomes, and small-business impacts, allowing a more complete view of how new housing interacts with neighborhood economies and climate goals.

Across the series, the central objective is to track and evaluate the real-world outcomes of Berkeley and State of California policies, laws, and regulations for both residents and businesses. The notebooks implement a full, transparent workflow from data → cleaning and enrichment → analysis → visualization and mapping → publication to SQL/Datasette, creating an extensible platform for evidence-based housing policy analysis.

Berkeley Civic Data Class - How a City Learns to See Itself Through Data
Berkeley HIgh School
UC Berkeley Data Science

💡 KEY SUCCESS FACTORS

1. Real Data: Students see their work matters in shaping who lives and works in their city

2. Public Output: Website creates accountability for the entire community

3. Create links among all educational levels: Berkeley High, UC Berkeley Data Science 8, UC Berkeley Data Science 100, Urban Research Centers, Terner Center, Urban

4. Community Connection: Guest speakers: stakeholders, city policy makers, city staff, neighbors--the fabric of our city

5. Scaffolded Learning: Week-by-week progression by all participants: students, teachers, city staff, city council, city economic leaders

6. Flexibility: Adapt to each city's data structures and open.gov policies, IT structure, advances in software and AI tools that transform how cities are managed

7. Celebration: Public presentations, media coverage by Berkleyside, Daily Cal, SF Chronicle, NY Times

🎓 BUILDING PERMIT PIPELINE: A CIVIC DATA EDUCATION FRAMEWORK 

📚 WHY THIS IS A PERFECT EDUCATIONAL PROJECT

Skills Students Learn:

Technical Skills:

• 📊 Data analysis (pandas, SQL)

• 🗺️ Geocoding and mapping

• 💾 Database design

• 🌐 Web development

• 📈 Data visualization

• 🤖 API usage

Civic Skills:

• 🏛️ Understanding local government

• 📋 Reading public records

• 🏗️ Urban planning concepts

• 📰 Data journalism

• 🗣️ Public communication

Real-World Impact:

• ✅ Addresses real community issues

• ✅ Creates public resource

• ✅ Students see their work used

• ✅ Builds portfolio projects

• ✅ Connects classroom to city hall

Why Berkeley Can Lead the State of California

What Success Looks Like:

For Students:

• Build technical portfolio

• Understand civic processes

• Create public resource

• College application material

For Teachers:

• Engaging project-based learning

• Real-world connections

• Interdisciplinary (CS + civics + math)

• Community partnerships

For Cities:

• Better informed residents

• Youth civic engagement

• Free data visualization

• Transparency tool

For Communities:

• Housing development tracking

• Accountability mechanism

• Planning input

• Data-driven advocacy