Memiro Project Explores Uses for Self-Tracked Data

Each of us create mounds of data every day—from what we eat, to how much time we exercise, to where we go, and beyond. Most of the data collected about us is passive data. Examples of such data include browsing history and location tracking. This data is objective, often unfiltered, and can be used to derive insights about users. However, because of its inactive form, this data is often not updated and cannot be utilized by the user for their own improvement. Worse, passive data collection raises concerns about data privacy and usage. Mozilla surveyed 20 apps for women’s healthcare and found that the majority collected large amounts of personal data and shared it with third parties.

Active or self-tracked data, however, allows users to have control over what is collected, what is actionable, and the insights a user can derive from it. Taking control of our own data and using data for the welfare of our communities are two pressing areas that need attention—and that is what ID’s 2022 Memiro project sought to do. 

Memiro included a summer design-build experience focused on wellbeing led by ID Assistant Professor of Design Zach Pino. Students elected to focus on self-tracked data, and through the course of the work developed a point of view on how personal data could enhance wellbeing. The Memiro team decided to focus on the needs of vulnerable populations, such as those dealing with memory decline, threats to community and personal safety, body dysmorphia (such as that from amputation or gender transition), and menstruation and fertility planning. 

Eschewing traditional persona and scenario-building approaches, the team focused on capturing rich data about individuals and specific communities. Garnering such data voluntarily from individuals helps build more inclusive and equitable futures in the long-term. 

Through their research, they proposed a critical sixth “V” to add to the commonly accepted 5 V’s of Big Data: Vulnerability. Moving forward with this proposition, they centered their solutions around the knowledge that vulnerable populations are disenfranchised twice by data-driven systems — they are not only unserved by contemporary digital systems, but their existence is invisible to systems that use data to drive future interventions and innovations.

The Memiro project is thus articulated through four distinct designs: Digital Legacy, led by Roxanne Hoffman (MDes + MBA 2022), Community-Driven Cartography led by Shinichiro Kuwahara (MDes 2021), Moving Alignment led by Jorge Martinez Arana (MDes + MBA 2023); and Personal Encoding led by Sandhini Ghodeshwar (MDes 2023). 

A memory repository aids in aging and end-of-life scenarios

Contemporary electronic devices allow people to theoretically remember anything with the click of a button. But the reality is: relying on digital tools has made memories harder to hold. Memory recall becomes even more of an issue for people experiencing cognitive decline or dementia. With populations aging worldwide and digital amnesia on the rise, we have the opportunity to consider how digital experiences can support memory making, legacy generation, and data distribution.  The Memiro Digital Legacy project seeks to adapt human behavior to digital information in order to preserve privacy, provenance, and, ultimately, to reduce the cognitive burden of defining a legacy. 

The project offers possibilities for organizing the way we want to be remembered. It does so by creating a memory repository that leverages our digital identity, which may include media like photos, videos, documents, and/or biodata. The project also provides tools for people to collect and reflect on past events. In this system, all self-tracked data about the self can be a source of memories. Users can securely interact with their memory data to categorize, compress, synthesize, and distribute their legacy with their privacy and access preferences in mind. 

An algorithmic framework for memory technology allows people to take control of their digital legacy with minimal cognitive effort. In addition to helping people remember their recent memories, this has the potential to improve memory care during aging and at the end of life. The digital experience takes memory data, either collected from personal electronics or wearable devices, processes the data through the legacy infrastructure, and creates interactive data representations over time. Memories are “captured” through collecting user-approved data such as images, videos, or even bio-signals. The memories are then categorized by sentiment, people or objects recognized via machine learning and other factors significant to the user. 

The next phase of the journey is recollection, where memories are resurfaced throughout the user’s life to update the importance of a memory over time. Memories that grow in significance are recollected more frequently, while ones that reduce in significance are shown less frequently. The frequency by which memories are resurfaced to the user could mimic spaced repetition systems (SRS), used by language learning systems, to help people remember their most valued memories. The frequency may also help with those who are working through traumatic memories and may benefit from memory reconsolidation, as directed by a mental health professional. When memories are surfaced during the recollection phase, the cumulative legacy becomes crystallized through the compression of the multiple memory objects into one. In other words, it becomes a digital equivalent of a personal time capsule. 

Users can share their legacy in multiple ways. Because the digital legacy is now composed of organized data, it can be filtered based on the receiving party. Legacies can be distributed to external parties such as healthcare providers to support medical research, estate planning to distribute assets among beneficiaries, organizations to commemorate culture or archive institutional knowledge, and loved ones who inherit intimate information. In addition to this, the legacy infrastructure considers the type of information and security access based on user preferences and how those may change after death. 

Throughout the experience, users can interact with their legacy as it evolves. Using the visual language of a ginkgo tree, a ginkgo branch could be a living projection in the home to convey the growth and pruning of a user’s legacy where leaves are data visualizations of individual memories. The intent of a projection is to bring data visualization in a living space, similar to how a child’s height is marked on a door jamb year over year.

Digital Legacy provides the structure on which to develop memory-sensitive technologies—the immersive experience of one’s own memory data may help users develop better understanding of their life and legacy as it’s developing. If the technology were expanded, the data processing could be implemented through blockchain systems which encrypt data, convert to semantic hashing, and create an immutable record. Additionally, AI-generated media could enable people to create stylized versions of their memories to convey context while preserving anonymity, allowing a richer depth of communication from memory to application. 

Today there are a handful of startups that provide tools to document family stories, record our digital behaviors and activities, and remind us of our past photos and media. But none considers the synthesis needed by the user to glean what is important from what is not. Digital Legacy not only enables that synthesis, but relies on the user’s behaviors to discern preference toward certain memories over others. Additionally it affords data portability and tangibility. While some users are aware of the “delete my data” or “export my data” functions on digital profiles, rarely do they understand or customize what is actually returned. The digital legacy system would enable users to distribute their legacy archive with different attributes based on the audience, such as full access of media for family and limited access of asset information for legal services. Digital legacies are made tangible through data visualization projections that users can interact with. With this system, users can not only document memories as they do today, with photos, videos, audio, but will simultaneously be preparing a digital legacy for their estate planners, medical providers, loved ones, and future generations.

Crowd-generated navigation system builds and protects communities

The Memiro Community-Driven Cartography project is a digital safety map defined, updated, and shared within a community. Users customize which datasets to incorporate into their map based on the communities they identify with. With a wearable device as the interface, users can gesture when they do or do not feel safe in their location to update the safety index and receive the safest route to their destination.

Safety is a big concern for everyone, especially vulnerable and historically marginalized people in unfamiliar locations. This project proposes a new form of social and cultural interaction in a city using self-tracked data for collective benefit. Today, self-tracking is primarily for the sake of individual knowledge. Contemporary data-driven systems don’t focus on groups or communities, but largely on nameless or faceless individuals who compose larger data sets. Encouraging people in different communities and cultures to contribute to safety maps would provide more trusted and customized safety information and help elevate the diverse safety requirements needed by communities under threat, ultimately helping to establish a more just society. 

As an iterative safety map application, Community-driven Cartography links to a wearable device in which users’ gestures contribute to a dynamic safety index displayed for their chosen communities. The application acts as a navigation tool that generates the shortest distance to a location based on the map’s safety zones. 

The mobile application onboarding process is composed of four steps:

1. Users choose a community, such as a specific gender identity, race, or geographical area.
2. Users define safety attributes. The app’s user interface reflects the user’s selections and enables them to make changes with sliders allowing for granular control. 
3. Users moderate the community, deciding how trustworthy the information they receive from members of the same community is. 
4. Users determine their gestures. These gestures will send signals with their wearable device. This function allows users to choose a culturally-specific gesture or cater to their unique abilities. It is also an option for users to create an original motion that aligns with their identity. The designated gesture will send a safe or unsafe signal when they walk around the city. 

The wearable is a band worn from the back of a hand to the wrist. Based on a combination of user inputs and selected safety databases, the backend system calculates a compound safety index displayed as a range-defined colored layer composed of hexagonal cells overlaid on the map. Users can adjust the color scheme to represent their community identity better or obfuscate the map’s meaning to others. The app algorithm generates fast and safe routes for the user that incorporate the road length and safety index.  With the volatility of crime, harassment, and abuse in cities, this product aims to help oppressed individuals share safety within their communities.

Trust in civic institutions is waning due to divisive  politics and questioned positions of authority. But in activist and grassroots contexts, individual experience on the ground is trusted. Community-Driven Cartography demonstrates how smart cities emanate from the citizens within them. This project aims to elevate the value of personal data, aggregating it to help social and cultural collectives. Expanding it could decentralize data usage and support marginalized peoples in regaining data ownership. While the project’s focus is safety mapping, it has the potential to facilitate social events or cultural hot spots to create a more vibrant urban life for people in each community. With these features, city governments and social sectors could leverage the anonymized data to develop infrastructures and services that better serve the needs of citizens, specifically those most in need of support.

Compared with similar services, such as safety map applications for women like Safe & the City, this project is unique because of the custom communities, discreet interaction, and real-time information. The safety map’s real-time feedback offers relevant information in moments of crisis and gives users peace of mind, building their confidence and building safe communities as they navigate the city.

Monitoring physical movement for honest reflection and appropriate action

Our bodies are in a constant state of change: growing, aging, and susceptible to being hurt by our environment. However, current ways of collecting and visualizing self-tracking data do not take this into account. Users are shown snapshots of their data in fitness and mobility apps without the consideration of how their bodies have changed. In fact, they have limited interaction with their data to explore how they have changed across time. Having such access and awareness becomes more critical for vulnerable populations, especially those facing body dysmorphia, amputations, gender transitions. Moving Alignment aims to show a potential new way of collecting data that takes the body’s dynamic context into account and enables interaction with bodily image and movement data in a more intimate and meaningful way.

To make better use of the data collected from our bodies, we must find alignment between the technology we use to collect it, the methods used to present the data, and how we interact with it to extract valuable insights to expand the understanding of our bodies. Smart watches and data dashboards can alter our perceptions of self, but they can also create walls for adoption to those with different body types and data literacy levels that the technology has not intentionally created inclusivity for. All of these aspects act to enable or discourage people seeking to understand themselves.  

The Memiro Moving Alignment project is a body-affirming, non-invasive option to capture and reflect on data about the body, and incorporates a supportive mind-body alignment interaction for the user. The goal is not to pass judgment, but to create opportunities for intuitive learning during moments of transition. This has the potential to open up body-focused self-tracking for underrepresented populations, advance unique and identity-affirming means of measuring one’s body and range of motion, and generate open dialogue about self-image. The product consists of a data-gathering process that employs wearable devices for data collection and computer vision, as well as an application for interactive data visualizations. 

For the data collection process, there are two approaches: 1) soft, flexible, and wearable electronics incorporated in a plastic-embedded textile, and 2) video-captured pose detection to reflect our bodies virtually. The former unites wearable electronics with form fitting textiles to mitigate the feeling of mismatch when using a personal device. The latter relies on gesture, posture, and body position to capture key points of the body, such as joints, allowing comparisons to be made as a user’s range of motion changes. Designing with a sensitivity to triggering images is paramount; the user can interact with their data visualization as a particle field instead of a photorealistic image if they choose. As users grow more comfortable and confident in their body image, the product can be adjusted to measure and display different attributes of themselves. Less invasive data collection and more meaningful interactions and takeaways from it could reshape the state of self-tracking and deepen our understanding of our own body and its movements.  

Today, by starting the discussion around how people collect and interact with their movement data, we ask ourselves if the data collected is really used with our best interests in mind, or if it’s mainly being used to satisfy the interests of the organizations that collect it. With rising concerns on privacy and data collection, this critical view—which takes into account both how our bodies, the source of data, evolve through time, and the many different types of bodies there are—has the potential to impact the way we develop future systems.  

Body measurement is rarely perfect. Data collection on body measurements is often invasive and judgmental due to the technology or necessary personnel performing the measurement. As users force themselves to fit the categories pre-defined for them, such as clothing sizes or crutch settings, they may feel alienated or oppressed. Moving Alignment presents a physical and visual data capture alternative that is body-affirming and minimally invasive. The system’s data visualizations support user interaction and exploration with their personal data, which can bring a new language for data interpretation while helping people feel aware and comfortable in their own skin.

Analog encoding protects sensitive information 

Personal Encoding is a guided experience to log, transform, and embed sensitive data in everyday objects. With an instruction manual and a kit of parts, the experience teaches individuals how to discreetly encode their own data to build self-knowledge and data literacy.

With the overturning of Roe v. Wade in the United States, abortion became a crime in many states and mHealth period-tracking apps, such as Flo or Cycles, were scrutinized for policies that could violate consumers’ privacy by making personal and sensitive data about period days, missed periods, predicted days of pregnancy, and abortion available to legal authorities. With this news, incrimination for making decisions on one’s own body became a reality. Personal Encoding returns the agency back to menstruating people through decentralizing and obscuring sensitive data. The act of tracking instills self-knowledge of one’s own body, which develops a personal sense of prediction. Instead of relying on internet communication technology, Personal Encoding employs manual data tracking and analysis through physically encoded forms, specifically jewelry beads.  

Memiro’s Personal Encoding acts as a critical thinking tool, questioning the times we live in and  and provoking audiences to think of a future where people prefer primitive technology (beading) over advanced tech (Big Data) to record, store, and use data.  

They are designed as a set of guided instructions that allows individuals to embed sensitive information into everyday objects. The product toolkit consists of an instruction manual in the form of a zine, a set of beads, and two sample jewelry pieces with their associated decoding patterns. The zine has basic instructions for mapping data points to objects, in this case, beads. It provides people with techniques to build their own data models for any data that they may want to track over any period of time.  

Personal Encoding is a personalized and accessible system. An individual can encode any kind of sensitive data discreetly. For example, data which might feel embarrassing and traumatic to share, such as suicidal thoughts, can be hidden within the symbol of a bead. Depending on an individual’s cultural, geographical, and political environment, one can decide the level of discretion that they embed into the coding structure. The user can represent individual tracking metrics in various bead colors, textures, or sizes depending on the number of parameters being measured. As a beading experience, it gives agency to its user to represent their culture by wearing beads or encompassing beading patterns belonging to their culture. Time can be encrypted based on user-specified time scales in days, hours, months, or years that do not need to follow a 28-day menstruation frequency.  

The system could expand to other physical objects, technological capabilities, and decentralized data processing. Activities such as embroidery, stitching, or collection hobbies can encode personal data once the user has a basic understanding of the data format. In an advanced stage, a mobile app experience could enable computer vision to securely read and decode bead patterns or derive new patterns of encryption. With edge computing and blockchain, computing can be distributed and anonymized across secure connections, bringing computation and data storage closer to the data source.

Leveraging edge technology and computer vision, one could extend a personal artifact to become a powerful tool. The beads could be scanned using computer vision and their meaning can be assigned on scanning, the encoding system can be stored locally via edge computing. In this way, the user’s data remains locally available to them on their device with minimal risk of a data breach or unwarranted sale. The future scope of Personal Encoding does not deny the skepticism around data, data capitalism, and surveillance, but provides an alternative means of harnessing the value of personal information in a secure and trusted context.

While similar physical period trackers exist in the market, their intent is not to maintain privacy but to enable young girls to understand their changing bodies. Often, they only keep track of one’s period and not other related symptoms, track according to a fixed cycle length, avoid sensitive topics such as abortion, and are limited in personalization. A critical difference between period tracking apps and the Personal Encoding is that this approach can be used to track any sensitive information in any format. The overall intent to build data literacy and self-knowledge is paramount in this project, but an afterthought in the current market.

Moving forward, those involved in the Memiro projects are hopeful that the areas they explored and the resulting methods they proposed might be adopted by organizations that cannot only understand the value of collecting data around vulnerability, but prioritize it.