Created at 12pm, Mar 27
Ms-RAGScience
0
BEYOND SIGHT: PROBING ALIGNMENT BETWEEN IMAGE MODELS AND BLIND V1
UsZgHqhInaqf899iUwDEnh9SZhpfNjvuy44BSeq4tfI
File Type
PDF
Entry Count
49
Embed. Model
jina_embeddings_v2_base_en
Index Type
hnsw

Jacob Granley, Galen Pogoncheff, Alfonso Rodil, Leili Soo, Lily Marie Turkstra, Lucas Gil Nadolskis , Arantxa Alfaro Saez, Cristina Soto Sanchez, Eduardo Fernandez Jover, Michael BeyelerABSTRACTNeural activity in the visual cortex of blind humans persists in the absence of visual stimuli. However, little is known about the preservation of visual representation capacity in these cortical regions, which could have significant implications or neural interfaces such as visual prostheses. In this work, we present a series ofanalyses on the shared representations between evoked neural activity in the primary visual cortex (V1) of a blind human with an intracortical visual prosthesis, and latent visual representations computed in deep neural networks (DNNs). In the absence of natural visual input, we examine two alternative forms of inducing neural activity: electrical stimulation and mental imagery. We first quantitatively demonstrate that latent DNN activations are aligned with neural activity measured in blind V1. On average, DNNs with higher ImageNet accuracy or higher sighted primate neural predictivity are more predictive of blind V1 activity. We further probe blind V1 alignment in ResNet-50 and propose a proof-of-concept approach towards interpretability of blind V1 neurons. The results of these studies suggest he presence of some form of natural visual processing in blind V1 during electrically evoked visual perception and present unique directions in mechanistically understanding and interfacing with blind V1.

The discovery of a significant, positive correlation between DNN-blind V1 alignment and DNNsighted neural alignment suggests the potential use of V1-aligned DNN models for studying visual 5 To appear at the ICLR 2024 Workshop on Representational Alignment (Re-Align) processing in both sighted and blind individuals. Interestingly, however, we observed that DNNblind V1 alignment was more correlated with DNN-sighted overall neural predictivity than with DNN-sighted V1 neural predictivity. This observation raises two critical questions: 1) whether electrically evoked visual perception in blind individuals activates different processing mechanisms than natural visual processing, and 2) whether cortical reorganization alters the functional role of V1 in the blind.
id: 22a3ffc19e85b73a7160a3cca04259d6 - page: 5
A notable limitation of the present study is that the dataset used for analysis contains a limited number of neural responses and visual percepts. Furthermore, our method relies on phosphene drawings for predicting neural activity. Even in this limited setting, well-aligned DNNs could inform more natural stimulation strategies for visual prostheses or deepen our understanding of neural representations in blindness. Moreover, innovative approaches such as topographic networks (Schrimpf et al., 2024) might predict neural activity without these drawings. Despite these challenges, our findings provide a valuable proof of concept, suggesting that existing techniques for assessing representational alignment could be extended and applied to understanding the visual cortex in blind humans. 6 To appear at the ICLR 2024 Workshop on Representational Alignment (Re-Align)
id: b93f4923e48dc78d57f776f96a4e77ab - page: 6
REFERENCES Anke Marit Albers, Peter Kok, Ivan Toni, H. Chris Dijkerman, and Floris P. de Lange. Shared representations for working memory and mental imagery in early visual cortex. Current biology: CB, 23(15):14271431, August 2013. ISSN 1879-0445. doi: 10.1016/j.cub.2013.05.065. Felix Bartsch, Gilava Hamuni, Vladimir Miskovic, Peter J. Lang, and Andreas Keil. Oscillatory brain activity in the alpha range is modulated by the content of word-prompted mental imagery. Psychophysiology, 52(6):727735, June 2015. ISSN 0048-5772. doi: 10.1111/psyp.12405. URL Pouya Bashivan, Kohitij Kar, and James J. DiCarlo. Neural population control via deep image ISSN 1095-9203. doi:
id: 239c302c0e41273d8b1629638e3bacb7 - page: 7
Science (New York, N.Y.), 364(6439):eaav9436, May 2019. 10.1126/science.aav9436. Michael Beyeler, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. Learning to see again: biological constraints on cortical plasticity and the implications for sight restoration technologies. Journal of Neural Engineering, 14(5):051003, August 2017. ISSN 1741-2552. doi: 10.1088/ 1741-2552/aa795e. URL Publisher: IOP Publishing. H. Burton. Visual Cortex Activity in Early and Late Blind People. The Journal of Neuroscience, 23(10):40054011, May 2003. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.23-10-04005.2003. URL
id: eb1b124a76d03b7804f74142d99a5a41 - page: 7
How to Retrieve?
# Search

curl -X POST "https://search.dria.co/hnsw/search" \
-H "x-api-key: <YOUR_API_KEY>" \
-H "Content-Type: application/json" \
-d '{"rerank": true, "top_n": 10, "contract_id": "UsZgHqhInaqf899iUwDEnh9SZhpfNjvuy44BSeq4tfI", "query": "What is alexanDRIA library?"}'
        
# Query

curl -X POST "https://search.dria.co/hnsw/query" \
-H "x-api-key: <YOUR_API_KEY>" \
-H "Content-Type: application/json" \
-d '{"vector": [0.123, 0.5236], "top_n": 10, "contract_id": "UsZgHqhInaqf899iUwDEnh9SZhpfNjvuy44BSeq4tfI", "level": 2}'