Zenan Li1, Zehua Liu 2, Yuan Yao 1, Jingwei Xu 1, Taolue Chen 3, Xiaoxing Ma 1, Jian Lü 1 1- State Key Lab of Novel Software Technology, Nanjing University, China2- Department of Mathematics, The University of Hong Kong, Hong Kong3- Department of Computer Science, Birkbeck, University of London, UKlizn@smail.nju.edu.cn, liuzehua@connect.hku.hk, t.chen@bbk.ac.uk, {y.yao,jingweix,xxm,lj}@nju.edu.cnABSTRACTRecent studies have explored the integration of logical knowledge into deep learning via encoding logical constraints as an additional loss function. However, existing approaches tend to vacuously satisfy logical constraints through shortcuts, failing to fully exploit the knowledge. In this paper, we present a new framework for learning with logical constraints. Specifically, we address the shortcut satisfaction issue byintroducing dual variables for logical connectives, encoding how the constraint is satisfied. We further propose a variational framework where the encoded logical constraint is expressed as a distributional loss that is compatible with the model’s original training loss. The theoretical analysis shows that the proposed approach bears salient properties, and the experimental evaluations demonstrate its superior performance in both model generalizability and constraint satisfaction.
In this paper, we have presented a new approach for better integrating logical constraints into deep neural networks. The proposed approach encodes logical constraints into a distributional loss that is compatible with the original training loss, guaranteeing monotonicity for logical entailment, significantly improving the interpretability and robustness, and avoiding shortcut satisfaction of the logical constraints at large. The proposed approach has been shown to be able to improve both model generalizability and logical constraint satisfaction. A limitation of the work is that we set the target distribution of any logical formula as the Dirac distribution, but further investigation is needed to decide when such setting could be effective and whether an alternative could be better. Additionally, our approach relies on the quality of the manually inputted logical formulas, and complementing it with automatic logic induction from raw data is an interesting future direction. 9
id: 4d9b81db631195f0065721ea5a18e78e - page: 9
Published as a conference paper at ICLR 2023 ACKNOWLEDGMENT We are thankful to the anonymous reviewers for their helpful comments. This work is supported by the National Natural Science Foundation of China (Grants #62025202, #62172199). T. Chen is also partially supported by Birkbeck BEI School Project (EFFECT) and an overseas grant of the State Key Laboratory of Novel Software Technology under Grant #KFKT2022A03. Yuan Yao (y.yaonju.edu.cn) and Xiaoxing Ma (xxmnju.edu.cn) are the corresponding authors.
id: 94c59a3013a9dde752bbe99c31ff1124 - page: 10
REFERENCES Leonard Adolphs. Non convex-concave saddle point optimization. Masters thesis, ETH Zurich, 2018. Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic probabilistic layers for neuro-symbolic learning. arXiv preprint arXiv:2206.00426, 2022. Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. Learning from rules generalizing labeled exemplars. In International Conference on Learning Representations, 2020. Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. 2017. Samy Badreddine, Artur dAvila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor
id: 16ef02ca0afd506d213d7d03ddce2e94 - page: 10
Artificial Intelligence, 303:103649, 2022. Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sbastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. In International Conference on Learning Representations, 2020. David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859877, 2017. Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. Ming-Wei Chang, Lev-Arie Ratinov, Nicholas Rizzolo, and Dan Roth. Learning and inference with constraints. In Proceedings of the 23rd national conference on Artificial intelligence, pp. 15131518, 2008. Xinshi Chen, Yufei Zhang, Christoph Reisinger, and Le Song. Understanding deep architectures with
id: 1b65f9ead6dac974d27141d1913d693f - page: 10