Methodology and Data Collection Instruments

Our work seeks to answer the following research questions:

1. How can digital/virtual learning experiences be designed and deployed such that they trigger interest in STEM? (design of triggers)
2. What features of informal learning experiences best frame science learning and encourage reengagement with content over time? (framing of learning)
3. What pedagogical strategies, as delivered by pedagogical agents, are most effective in promoting STEM interest and learning? (promoting interest)
4. How can a technology infrastructure be used to monitor and track changes in STEM interest over time, specifically for groups who are underrepresented in STEM? (monitoring of interest)

Avenues for Data Collection

The WHIMC server was originally conceived as a destination for players across the internet connecting from home as much as it was a host for camps and workshops. As such it was designed with a Quests system and NPC’s to guide the experience and provide a degree of self-guided learning. Eventually we shifted focus to rely on the WHIMC more as a teacher-leveraged learning environment for collaboration and problem-solving.

Instrument to Question and Data Mapping

As such we’ve assembled an array of research instruments to collect data to approach research questions from an affective, behavioral and cognitive perspective:

Instrument	Research Question	Data and Analysis
Surveys	1, 2, 3	Astronomy knowledge, STEM interests and Minecraft experience (latest/refined based on the most statistically relevant items), AI learning survey (2022) (2023)
Log data	1, 2, 3, 4	In-game observations and science tools usage, exploration and NPC/AI interaction patterns
Self Explanations	4	Conceptual and content knowledge checks in the form of self explanations (scoring) derived from Renninger and Hidi (2022), previously ICAN (2021 prior)
Exoplanet drawings	2	Content analysis evidence of scientific knowledge and reasoning
Fieldnotes, audio/screen capture	1, 2, 3, 4	What if questions, STEM career interest, qualtiative measures of engagement, observation of in-game AI agent use, group problem-solving discussion for Mynoa mystery puzzle
Interviews	1, 2, 4	Astronomy and Minecraft interest and knowledge (protocol available)
Mars Habitats	3, 4	Validated content analysis schema of collaborative (participants in teams) habitat creations on mars; this forms a comparison basis for an AI pedagogical agent that uses a combination of screen capture and block data to analyze and issue feedback

Log Data Breakdown

Core Protect is a plugin that keeps track of the status of all blocks, commands and interactions with the game, as tied to users in sessions. It can be stored in a sqlite database or as mysql, we typically use the latter, unless it is on a local server. It includes many tables all with the prefix “co_”

Quests is a plugin that gives users optional tasks they can complete as part of the game. There is a quest journal (book/tablet) that lists them and they can include objectives like talking to NPC’s or going to specific locations to take measures. We use it primarily as a single player (at-home) learning framework. It can be stored in separated yml (text like xml) files that are more granular or as part of the sql database. Quests can make use of our science tools and observation plugins to use them as objectives. We mostly do not use it for research since we recycle user accounts between camps and many users struggle to read chat text, which is the primary format information is conveyed for quests. Quests also vary considerably in their scope and difficulty and can be done in various orders. We tested several iterations storing more granular quest data in yml files and less-specific data in the database, since it could be linked to other performance measures more easily there. Ultimately instead of relying on Quests for research data they’ve become another kind of teaching aid and potential driver of interest for achievement-oriented participants. A paper-based hand-out set can provide some of the same functions as quests for younger participants, as it’s less information and teachers can point at it as a checklist.

WHIMC has a collection of plugins – a teacher agent, builder agent, observations, science tools and player movement tracker/recommender. They have a variety of supporting tables and several collect data.

Besides the above SQL databases the server has log data for all chat exchanges and commands issued as well as config files for many plugins that confer player statuses, such as their last location or which regions or actions they’re allowed to access (permissions).

Currently Used Tables

• co_block – all block modifications data; used for habitat analysis PA, includes:
  • time of interaction (unix 10 digit)
  • world ID it happened on (hub, kepler, etc)
  • coordinates of the block (x, y, z)
  • type of block (iron, sand, etc)
  • type of interaction (destroy, place, etc)
  • state status (see blockdata_map)
  • if it has been rolled back (undo)
• whimc_action_physical
  • user interacting with container (UUID, username)
  • world it happened on (hub, kepler, etc)
  • coordinates where recorded (x, y, z)
  • time stamp (unix 10 digit)
  • type of interaction (pressure plate, tripwire, and redstone ore interactions)
• whimc_containers
  • containers include chests, barrelbots, basically anything you can put stuff in
  • user interacting with container (UUID, username)
  • world it happened on (hub, kepler, etc)
  • coordinates where recorded (x, y, z)
  • time stamp (unix 10 digit)
  • which inventory slots changed in the container
• co_session – a giant table of all player sessions, including:
  • time stamp (unix 10 digit)
  • user for that session
  • world ID it happened on (hub, kepler, etc)
  • coordinates where recorded (x, y, z)
  • action – in this case starting or stopping the session
• co_username_log – UUID’s for all players; this is a unique incomprehensible number that’s like a SSN for each user; includes their in-game nickname and time of first connection
• co_world – ID’s for all worlds on the server, like Mynoa or Hub
• whimc_observations – floating text attached to invisible armor stands conveying what players type; can also be used with a templating system that encourages descriptive, comparative, inferential and inquiry observations; includes:
  • time of observation (unix 10 digit)
  • UUID and username
  • world (like Cancri’s cold side or Lunar Crater)
  • coordinates where the observation was made and the orientation of what the player was looking at when it was made; making it possible to teleport to the observation to see exactly what they were looking at when they made it
  • the text of the observation
  • if it is still active (visible) as they disappear after a few days
  • expiration time when it went invisible
  • category for templated observations, such as descriptive or inferential
  • observation_color_stripped is the raw observation text that can be parsed more easily by external data processing
• whimc_player_positions is a tremendously huge (7 million entry and growing) log of intervals of player locations that feeds into several plugins; includes
  • coordinates (x, y, z)
  • world (hub, rocket launch, etc)
  • biome (type of environment such as desert or ocean)
  • username and UUID
  • time of location recording
• whimc_sciencetools is a listing of all science tool uses by player
  • time of observation (unix 10 digit)
  • UUID and username
  • world (like Cancri’s cold side or Lunar Crater)
  • coordinates where the tool was used
  • which tool was used and what measure it relayed

Potentially Important Tables

• co_blockdata_map – possible block states, such as a trap door being open or the age of wheat
• co_chat – a log of all chat (not commands) from participants
• co_entity – all entities, which are different than blocks; can include things like mobs (creatures), signs, armor stands, map/item frames and more; has a time stamp and an array of information about the entity – this array includes characteristics like invisibility, knockback resistance, health, following range and other similar behavior data
• co_material – ID’s for material types, such as glass or snow_block
• co_sign – data on all sign entities on the server; previously used for observations (2018) but now really just for labeling purposes
• co_skull – all “minecraft head” blocks – that can be “skinned” (decorated via pixel art) to look various ways, like computers or cabbage
• co_user – UUID’s for all entities; this is a unique incomprehensible number that’s like a SSN for each user; includes their in-game nickname and time of first connection; also Minecraft entity names like “Polar_Bear”
• quests_players – UUID’s for all players that have done some or all of any quest
• quests_player_completedquests – which quests each player (by UUID) has completed
• quests_player_currentquests – quests players (by UUID) are on currently
• quests_player_questdata – specific data about a given quest by player (UUID) like how many items they’ve collected, locations reached or custom objectives like observations made
• whimc_agents – for use with teacher agents, mostly tracking participant interactions with the PA such as time of interaction, player UUID/username, command issued like creating or renaming, what they’ve named their PA and which skin (appearance) they’ve selected
• whimc_build_templates – block arrangement templates (schematics) saved to the builde helper agent, which parrots player build actions
• whimc_dialogue – a record (time, world, player) of player interaction “dialogue” with the teacher PA, including observation ratings, quests completed, science tools uses, exploration score, science topics exhibited determined via text mining
• whimc_dialogue_interaction – a record (time, world) of when the teacher PA talks to a player (UUID/username) for their tags, edits or guiding them
• whimc_dialog_science – a record (time, user, world) of users asking the PA for science information
• whimc_progress – scoring of player observations, science tools, explorations and quest accomplishments along with a total; players can access this information talking to the teacher agent to get a sense of how well they’re doing from a learning standpoint
• whimc_dialogue_builder_interaction – tracks how students use the builder agent to hopefully correlate interaction types with building outcomes
• whimc_progress_commands – record of times players query the teacher agent for their progress
• whimc_skills is player use of observation templates – how many times they did analogies, comparisons, descriptions or inferences
• whimc_tags where and when players interacted with the agent about something in a given recorded world

Mostly Irrelevant Tables

• co_art_map – Minecraft’s built-in decorative paintings
• co_command – a log of all world edit commands issued by participants, so just designers creating the interfaces are represented here
• co_container – includes coordinates and contents for storages chests, shulker boxes, etc (our whimc_containers table provides more information)
• co_database_lock – database lock status
• co_entity_map – ID’s mapped to standard mob (creature) entities like axolotls, skeletons and so on
• co_item – not totally sure what this is, but it has a time stamp, user, coordinates, type of item, data about the item, amount of item, action for it and if it’s been rolled back; quite possibly player inventory but it has coordinates which wouldn’t make sense for that
• co_version – core protect version history
• quests_player_redoablequests – quests players have done multiple times; we don’t have any of these

Player Exploration Metrics

We have two ways of rating player explorations automatically.

1. Total exploration score via the path displayer, which uses location information to show player maps and observations made (first image above). We currently use this metric but it fails to capture the difference between players really exploring and players goofing around trying to exploit bugs to escape the map or playing hide and seek.
2. Exploration scores can now be weighted (second image above) based on locations coded into a configuration file that may correspond with characters (NPC’s), important points of observation (science characteristics) or quest objectives. These will be based on “signals” derived from our research indicating interest as well as written into curriculum via our teacher guides.

Data-Driven Exploratory Case Studies

Several other studies have included:

• A data science class project about mapping player interests to game behaviors by Brian Guerrero
• Various explorations into correlations connecting different player behaviors and learning outcomes primarily using the database (Janelle Domantay, Matt Gadbury, Chris Palaguchi)