3-pre-Leibniz Algebras, Deformations and Cohomologies of Relative Rota-Baxter Operators on 3-Leibniz Algebras

Abstract

In this paper, first we introduce the notions of 3-pre-Leibniz algebras and relative Rota-Baxter operators on 3-Leibniz algebras. We show that a 3-pre-Leibniz algebra gives rise to a 3-Leibniz algebra and a representation such that the identity map is a relative Rota-Baxter operator. Conversely, a relative Rota-Baxter operator naturally induces a 3-pre-Leibniz algebra. Then we construct a Lie 3-algebra, and characterize relative Rota-Baxter operators as its Maurer-Cartan elements. Consequently, we obtain the $L_\infty$-algebra that controls deformations of relative Rota-Baxter operators on 3-Leibniz algebras. Next we define the cohomology of relative Rota-Baxter operators on 3-Leibniz algebras and show that infinitesimal deformations of a relative Rota-Baxter operator are classified by the second cohomology group. Finally, we construct an $L_\infty$-algebra whose Maurer-Cartan elements are relative Rota-Baxter 3-Leibniz algebra structures, and define the cohomology of relative Rota-Baxter 3-Leibniz algebras.

1 Introduction

In this paper, we mainly study deformations and cohomologies of relative Rota-Baxter operators on 3-Leibniz algebras and relative Rota-Baxter 3-Leibniz algebras respectively.

1.1 3-Leibniz Algebras

Leibniz algebras can be regarded as a non-skew-symmetric version of Lie algebras. The class of n-Leibniz algebras as a natural generalization of Leibniz algebras to higher arities was introduced by Casas et al. [9]. Furthermore, an n-Leibniz algebra combined with skew-symmetry is an n-Lie algebra (also called Filippov algebras or Nambu algebras) [19]. In recent years, both 3-Lie algebras and 3-Leibniz algebras have extensive applications in mathematical physics, e.g. in [10], the authors used Gröbner bases to present the algorithm in order to test the given multiplication table corresponding to an n-Leibniz algebra. Cherkis and Sämann constructed the three-dimensional $N=2$ superconformal Chern-Simons theories by imposing extra conditions on the 3-bracket [11]. On this foundation, in [17], the authors studied metric 3-Leibniz algebras and the hermitian 3-algebras of Bagger-Lambert [1], which are closely related to $N=6$ and $N=5$ superconformal Chern-Simons theories. In [7], the authors studied the structure of split 3-Leibniz algebras. See the review [15] for more details.

1.2 Relative Rota-Baxter Operators

The notion of Rota-Baxter operators on algebras originated from the work of G. Baxter in the study of fluctuation theory in probability [6]. It has been used in many fields of mathematical physics, including the Connes-Kreimer’s algebraic approach to the renormalization in perturbative quantum field theory [12]. Rota-Baxter operators are also closely related to splitting of operads [2, 29]. In the Lie algebra context, the more general notion of a relative Rota-Baxter operator (originally called O-operator) was introduced by Kupershmidt [25] to better understand the classical Yang-Baxter equation. See the book [23] for more details and applications about Rota-Baxter operators. Relative Rota-Baxter operators on Leibniz algebras was studied in [34]. Recently, in [3], the authors introduced the concept of Rota-Baxter operators on 3-Lie algebras and established their relationship with differential operators. The notion of relative Rota-Baxter operators on 3-Lie algebras was introduced in [4] for the purpose of studying solutions to the classical Yang-Baxter equation for 3-Lie algebras.

1.3 Deformations and Cohomologies

The deformation theory of algebraic structures originated from Gerstenhaber for associative algebras [21]. Inspired by this theory, Nijenhuis and Richardson further extended the work to Lie algebras [28]. In [18], the author studied the deformation of 3-Leibniz algebras in terms of the cohomology theory of its associated Leibniz algebra. See [27, 32] for more details about deformations of n-Leibniz (Lie)-algebras. Cohomology theory is an important invariant of algebraic structures and can be used to study deformations and extensions of algebraic structure. Loday and Pirashvili studied the cohomology theory and the universal envelo** algebra structure of Leibniz algebras [26]. Rotkiewicz introduced graded Lie brackets on the space of cochains of n-Leibniz algebras and described an n-Leibniz algebra structure as a canonical structure [30]. Recently, the cohomology and deformation theories of relative Rota-Baxter operators on Lie algebras, associative algebras, Leibniz algebras and 3-Lie algebras were established in [14, 33,34,35], respectively.

1.4 Approaches and Main Results

Motivated by the widespread application of Rota-Baxter algebras, 3-Leibniz algebras, cohomology and deformation theories, it is natural to study relative Rota-Baxter operators on 3-Leibniz algebras. First we introduce the notion of a 3-pre-Leibniz algebra and relative Rota-Baxter operators on 3-Leibniz algebras. We prove that there is a 3-pre-Leibniz algebra structure on V as the underlying algebraic structure of a relative Rota-Baxter operator on a 3-Leibniz algebra. Moreover, a 3-pre-Leibniz algebra naturally gives rise to a new 3-Leibniz algebra such that the identity map is a relative Rota-Baxter operator. Next, we construct an $L_{\infty }$-algebra using higher derived brackets whose Maurer-Cartan elements are relative Rota-Baxter operators on 3-Leibniz algebras. Maurer-Cartan elements of the twisted $L_{\infty }$-algebra correspond precisely to deformations of the given relative Rota-Baxter operator. Moreover, we also set up a cohomology theory for relative Rota-Baxter operators on 3-Leibniz algebras and apply it to control infinitesimal deformations of relative Rota-Baxter operators. A relative Rota-Baxter 3-Leibniz algebra consists of a 3-Leibniz algebra, its representation and a relative Rota-Baxter operator. Finally, we construct the corresponding cohomology theory of relative Rota-Baxter 3-Leibniz algebras and show that infinitesimal deformations of relative Rota-Baxter 3-Leibniz algebras are classified by the second cohomology group.

1.5 Outline of the Paper

In Sect. 2, we introduce the notions of 3-pre-Leibniz algebras and relative Rota-Baxter operators on 3-Leibniz algebras. We show that a 3-pre-Leibniz algebra gives rise to a 3-Leibniz algebra and a representation such that the identity map is a relative Rota-Baxter operator. Conversely, a relative Rota-Baxter operator naturally induces a 3-pre-Leibniz algebra. In Sect. 3, we construct a Lie 3-algebra and show that relative Rota-Baxter operators can be characterized by its Maurer-Cartan elements. In addition, a given relative Rota-Baxter operator gives rise to a twisted $L_{\infty }$-algebra whose Maurer-Cartan elements controls its deformations. In Sect. 4, we introduce a cohomology theory of relative Rota-Baxter operators on 3-Leibniz algebras. As an application, infinitesimal deformations of relative Rota-Baxter operators are studied. In Sect. 5, we characterize relative Rota-Baxter 3-Leibniz algebras as Maurer-Cartan elements in a certain $L_{\infty }$-algebra, and establish the corresponding cohomology and deformation theories respectively.

In this paper, we work over an algebraically closed field $\mathbb K$ of characteristic 0.

2 3-pre-Leibniz Algebras and Relative Rota-Baxter Operators on 3-Leibniz Algebras

In this section, we introduce the notions of relative Rota-Baxter operators on 3-Leibniz algebras and 3-pre-Leibniz algebras. A 3-pre-Leibniz algebra gives rise to a 3-Leibniz algebra and a representation on itself. We show that a relative Rota-Baxter operator induces a 3-pre-Leibniz algebra. Thus 3-pre-Leibniz algebras can be viewed as the underlying algebraic structures of relative Rota-Baxter operators on 3-Leibniz algebras. First, we recall some basic results involving representations and cohomologies of 3-Leibniz algebras.

Definition 2.1

[9] A 3-Leibniz algebra is a vector space $\mathcal {L}$ equipped with a linear map $[\cdot ,\cdot ,\cdot ]_{\mathcal {L}}:\otimes ^3\mathcal {L}\longrightarrow \mathcal {L}$ such that

$$\begin{aligned} {[x_{1},x_{2},[x_{3},x_{4},x_{5}]_{\mathcal {L}}]}_{\mathcal {L}}=[[x_{1},x_{2},x_{3}]_{\mathcal {L}},x_{4},x_{5}]_{\mathcal {L}}+[x_{3},[x_{1},x_{2},x_{4}]_{\mathcal {L}},x_{5}]_{\mathcal {L}}+[x_{3},x_{4},[x_{1},x_{2},x_{5}]_{\mathcal {L}}]_{\mathcal {L}}, \end{aligned}$$

(1)

for all $x_{1},x_{2},x_{3},x_{4},x_{5} \in \mathcal {L}$.

Definition 2.2

[8] A representation of a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ is a quadruple (V; l, m, r), where V is a vector space, $l, m, r:\otimes ^{2}\mathcal {L}\longrightarrow \mathfrak {gl}(V)$ are linear maps, such that for all $x_{i} \in \mathcal {L}, 1\le i\le 4$, the following equalities hold:

$$\begin{aligned} l(x_{1},x_{2})l(x_{3},x_{4})= & {} l([x_{1},x_{2},x_{3}]_{\mathcal {L}},x_{4})+l(x_{3},[x_{1},x_{2},x_{4}]_{\mathcal {L}})+l(x_{3},x_{4})l(x_{1},x_{2}), \end{aligned}$$

(2)

$$\begin{aligned} l(x_{1},x_{2})m(x_{3},x_{4})= & {} m([x_{1},x_{2},x_{3}]_{\mathcal {L}},x_{4})+m(x_{3},[x_{1},x_{2},x_{4}]_{\mathcal {L}})+m(x_{3},x_{4})l(x_{1},x_{2}),\end{aligned}$$

(3)

$$\begin{aligned} l(x_{1},x_{2})r(x_{3},x_{4})= & {} r([x_{1},x_{2},x_{3}]_{\mathcal {L}},x_{4})+r(x_{3},[x_{1},x_{2},x_{4}]_{\mathcal {L}})+r(x_{3},x_{4})l(x_{1},x_{2}),\end{aligned}$$

(4)

$$\begin{aligned} m(x_{1},[x_{2},x_{3},x_{4}]_{\mathcal {L}})= & {} r(x_{3},x_{4})m(x_{1},x_{2})+m(x_{2},x_{4})m(x_{1},x_{3})+l(x_{2},x_{3})m(x_{1},x_{4}),\end{aligned}$$

(5)

$$\begin{aligned} r(x_{1},[x_{2},x_{3},x_{4}]_{\mathcal {L}})= & {} r(x_{3},x_{4})r(x_{1},x_{2})+m(x_{2},x_{4})r(x_{1},x_{3})+l(x_{2},x_{3})r(x_{1},x_{4}). \end{aligned}$$

(6)

Example 2.3

Let $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ be a 3-Leibniz algebra. Define linear maps $L, M, R:\otimes ^{2}\mathcal {L}\rightarrow \mathfrak {gl}(\mathcal {L})$ by

$$\begin{aligned} L(x,y)z=[x,y,z]_{\mathcal {L}},\quad M(x,y)z=[x,z,y]_{\mathcal {L}},\quad R(x,y)z=[z, x, y]_{\mathcal {L}},\quad \forall x,y,z \in \mathcal {L}. \end{aligned}$$

Then $(\mathcal {L};L,M,R)$ is a representation of $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$, which is called the regular representation.

Let (V; l, m, r) be a representation of the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}}).$ Define a trilinear bracket operation $[\cdot ,\cdot ,\cdot ]_{\mathcal {L}\oplus V}:\otimes ^3(\mathcal {L}\oplus V)\rightarrow \mathcal {L}\oplus V$ by

$$\begin{aligned} {[}x_1+u_1,x_2+u_2,x_3+u_3]_{\mathcal {L}\oplus V}=[x_1,x_2,x_3]_{\mathcal {L}}+l(x_1,x_2)u_3+m(x_1,x_3)u_2+r(x_2,x_3)u_1, \end{aligned}$$

(7)

where $x_i \in \mathcal {L}, u_i \in V, 1\le i\le 3$. The following result is well known.

Lemma 2.4

With the above notations, $(\mathcal {L}\oplus V,[\cdot ,\cdot ,\cdot ]_{\mathcal {L}\oplus V})$ is a 3-Leibniz algebra. This 3-Leibniz algebra is called the semidirect product 3-Leibniz algebra, and denoted by $\mathcal {L}\ltimes _{l,m,r} V$.

Now we introduce a new algebraic structure: 3-pre-Leibniz algebras.

Definition 2.5

A 3-pre-Leibniz algebra is a vector space A equipped with three ternary products $[\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}$ and $[\![ {\cdot ,\cdot ,\cdot }]\!]: \otimes ^3 A \rightarrow A$ such that for all $x_i\in A,1\le i\le 5,$ the following equalities hold:

$$\begin{aligned} {[}x_1,x_2,[x_3,x_4,x_5]]= & {} [[x_1,x_2,x_3]_{A},x_4,x_5]+[x_3,[x_1,x_2,x_4]_A,x_5]+[x_3,x_4,[x_1,x_2,x_5]], \end{aligned}$$

(8)

$$\begin{aligned} {[}x_1,x_2,\{x_3,x_5,x_4\}]= & {} \{[x_1,x_2,x_3]_{A},x_5,x_4\}+\{x_3,[x_1,x_2,x_5],x_4\}+\{x_3,x_5,[x_1,x_2,x_4]_A\},\end{aligned}$$

(9)

$$\begin{aligned} {[}x_1,x_2,[\![ {x_5,x_3,x_4}]\!]]= & {} [\![ {[x_1,x_2,x_5],x_3,x_4}]\!] +[\![ {x_5,[x_1,x_2,x_3]_A,x_4}]\!] +[\![ {x_5,x_3,[x_1,x_2,x_4]_A}]\!] ,\end{aligned}$$

(10)

$$\begin{aligned} \{x_1,x_5,[x_2,x_3,x_4]_A\}= & {} [\![ {\{x_1,x_5,x_2\},x_3,x_4}]\!] +\{x_2,\{x_1,x_5,x_3\},x_4\}+[x_2,x_3,\{x_1,x_5,x_4\}], \end{aligned}$$

(11)

$$\begin{aligned} \quad [\![ {x_5,x_1,[x_2,x_3,x_4]_A}]\!]= & {} [x_2,x_3,[\![ {x_5,x_1,x_4}]\!] ]+\{x_2,[\![ {x_5,x_1,x_3}]\!] ,x_4\} +[\![ {[\![ {x_5,x_1,x_2}]\!] ,x_3,x_4}]\!] , \end{aligned}$$

(12)

where $[\cdot ,\cdot ,\cdot ]_A$ is defined by

$$\begin{aligned} {[}x,y,z]_A=[x,y,z]+\{x,y,z\}+[\![ {x,y,z}]\!] ,\quad \forall x,y,z\in A. \end{aligned}$$

(13)

Remark 2.6

In [4], Bai-Guo-Sheng introduced the notion of a 3-pre-Lie algebra which is the underlying algebraic structure of a relative Rota-Baxter operator (O-operator) on a 3-Lie algebra. In our definition of a 3-pre-Leibniz algebra $(A, [\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}, [\![ {\cdot ,\cdot ,\cdot }]\!] )$ for all $x,y,z\in A,$ if the ternary product $[\cdot ,\cdot ,\cdot ]:\otimes ^3 A \rightarrow A$ satisfies $[x,y,z]=-[y,x,z],$ $\{x,y,z\}=[z,x,y]$ and $[\![ {x,y,z}]\!] =[y,z,x]$, then $(A, [\cdot ,\cdot ,\cdot ])$ becomes a 3-pre-Lie algebra. So the 3-pre-Leibniz algebras are natural generalizations of 3-pre-Lie algebras introduced by Bai-Guo-Sheng. To be consistent with the terminology given in [4], we use the terminology of "3-pre-Leibniz algebra". On the one hand, from the perspective of splittings of operads [29], this algebraic structure could also be called pre-3-Leibniz algebra. Nevertheless, this algebraic structure serves as both the splitting of 3-Leibniz algebra and the 3-ary generalization of pre-Leibniz algebras.

Theorem 2.7

Let $(A, [\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}, [\![ {\cdot ,\cdot ,\cdot }]\!] )$ be a 3-pre-Leibniz algebra. Then

• (i) $(A, [\cdot ,\cdot ,\cdot ]_A)$ is a 3-Leibniz algebra, where the 3-Leibniz bracket $[\cdot ,\cdot ,\cdot ]_A$ is defined by (13).

• (ii) $(A;\tilde{L},\tilde{M},\tilde{R})$ is a representation of the 3-Leibniz algebra $(A, [\cdot ,\cdot ,\cdot ]_A)$, where the representation $\tilde{L},\tilde{M},\tilde{R}:\otimes ^{2}A\rightarrow \mathfrak {gl}(A)$ are defined by

  $$\begin{aligned} \tilde{L}(x,y)z=[x,y,z],\quad \tilde{M}(x,y)z=\{x,z,y\},\quad \tilde{R}(x,y)z=[\![ {z, x,y}]\!] ,\quad \forall ~x,y,z\in A. \end{aligned}$$

  (14)

Proof

(i) For all $x_1,x_2,x_3,x_4,x_5 \in A$, by (8) - (12), we have

$$\begin{aligned}{} & {} [[x_1,x_2,x_3]_A,x_4,x_5]_A+[x_3,[x_1,x_2,x_4]_A,x_5]_A+[x_3,x_4,[x_1,x_2,x_5]_A]_A\\{} & {} \qquad =[[x_1,x_2,x_3]_A,x_4,x_5]+\{[x_1,x_2,x_3]_A,x_4,x_5\}+[\![ {[x_1,x_2,x_3]_A,x_4,x_5}]\!] \\{} & {} \qquad +[x_3,[x_1,x_2,x_4]_A,x_5]+\{x_3,[x_1,x_2,x_4]_A,x_5\}+[\![ {x_3,[x_1,x_2,x_4]_A,x_5}]\!] \\{} & {} \qquad +[x_3,x_4,[x_1,x_2,x_5]_A]+\{x_3,x_4,[x_1,x_2,x_5]_A\}+[\![ {x_3,x_4,[x_1,x_2,x_5]_A}]\!] \\{} & {} \quad =[[x_1,x_2,x_3]_A,x_4,x_5]+\{[x_1,x_2,x_3]_A,x_4,x_5\}+[\![ {[x_1,x_2,x_3],x_4,x_5}]\!] \\{} & {} \qquad +[\![ {\{x_1,x_2,x_3\},x_4,x_5}]\!] +[\![ {[\![ {x_1,x_2,x_3}]\!],x_4,x_5}]\!]+[x_3,[x_1,x_2,x_4]_A,x_5]\\{} & {} \qquad +\{x_3,[x_1,x_2,x_4],x_5\}+\{x_3,\{x_1,x_2,x_4\},x_5\}+\{x_3,[\![ {x_1,x_2,x_4}]\!] ,x_5\}\\{} & {} \qquad +[\![ {x_3,[x_1,x_2,x_4]_A,x_5}]\!] +\{x_3,x_4,[x_1,x_2,x_5]_A\}+[\![ {x_3,x_4,[x_1,x_2,x_5]_A}]\!] \\{} & {} \qquad +[x_3,x_4,[x_1,x_2,x_5]]+[x_3,x_4,\{x_1,x_2,x_5\}]+[x_3,x_4,[\![ {x_1,x_2,x_5}]\!] ]\\{} & {} \quad =[x_1,x_2,[x_3,x_4,x_5]]+[x_1,x_2,\{x_3,x_4,x_5\}]+[x_1,x_2,[\![ {x_3,x_4,x_5}]\!] ]\\{} & {} \qquad +\{x_1,x_2,[x_3,x_4,x_5]_A\}+[\![ {x_1,x_2,[x_3,x_4,x_5]_A}]\!] \\{} & {} \quad =[x_1,x_2,[x_3,x_4,x_5]_A]+\{x_1,x_2,[x_3,x_4,x_5]_A\}+[\![ {x_1,x_2,[x_3,x_4,x_5]_A}]\!] \\{} & {} \quad =[x_1,x_2,[x_3,x_4,x_5]_A]_A, \end{aligned}$$

which implies that $(A, [\cdot ,\cdot ,\cdot ]_A)$ is a 3-Leibniz algebra.

(ii) We can deduce that $\tilde{L},\tilde{M},\tilde{R}$ satisfy (2)–(6) according to (8)–(12) respectively. $\square$

Definition 2.8

Let $(A, [\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}, [\![ {\cdot ,\cdot ,\cdot }]\!] )$ be a 3-pre-Leibniz algebra. The 3-Leibniz algebra $(A, [\cdot ,\cdot ,\cdot ]_A)$ is called the sub-adjacent 3-Leibniz algebra, and the 3-pre-Leibniz algebra $(A, [\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}, [\![ {\cdot ,\cdot ,\cdot }]\!] )$ is called a compatible 3-pre-Leibniz algebra of the 3-Leibniz algebra $(A, [\cdot ,\cdot ,\cdot ]_A)$.

Now we introduce the notion of (relative) Rota-Baxter operators on 3-Leibniz algebras, which are closely related to 3-pre-Leibniz algebras.

Definition 2.9

• (i) Let $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ be a 3-Leibniz algebra. A linear operator $B:\mathcal {L}\rightarrow \mathcal {L}$ is called a Rota-Baxter operator if

  $$\begin{aligned} {[}Bx,By,Bz]_{\mathcal {L}}=B\Big ([Bx,By,z]_{\mathcal {L}}+[Bx,y,Bz]_{\mathcal {L}}+[x,By,Bz]_{\mathcal {L}}\Big ), \quad \forall x, y, z \in \mathcal {L}. \end{aligned}$$

  (15)

  Moreover, a 3-Leibniz algebra $\mathcal {L}$ with a Rota-Baxter operator B is called a Rota-Baxter 3-Leibniz algebra, which is denoted by $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),B).$

• (ii) Let (V; l, m, r) be a representation of the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$. A linear operator $B:V\rightarrow \mathcal {L}$ is called a relative Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to the representation (V; l, m, r) if B satisfies

  $$\begin{aligned}{}[Bu,Bv,Bw]_{\mathcal {L}}=B\Big (l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u\Big ), \quad \forall u, v, w \in V. \end{aligned}$$

  (16)

  Moreover, the triple $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),(V;l,m,r),B)$ is called a relative Rota-Baxter 3-Leibniz algebra.

Note that a Rota-Baxter operator on a 3-Leibniz algebra is a relative Rota-Baxter operator with respect to the regular representation. A 3-pre-Leibniz algebra naturally gives rise to a relative Rota-Baxter operator.

Proposition 2.10

Let $(A, [\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}, [\![ {\cdot ,\cdot ,\cdot }]\!] )$ be a compatible 3-pre-Leibniz algebra of the 3-Leibniz algebra $(A,[\cdot ,\cdot ,\cdot ]_A)$. Then ${\textrm{Id}}:A\rightarrow A$ is a relative Rota-Baxter operator on the 3-Leibniz algebra $(A,[\cdot ,\cdot ,\cdot ]_A)$ with respect to the representation $(A;\tilde{L},\tilde{M},\tilde{R})$ given in Theorem 2.7.

Next, we consider the following 3-Leibniz algebra which was introduced in [10].

Example 2.11

Consider the 3-dimensional 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ given with respect to a basis $\{e_1,e_2,e_3\}$ by

$$\begin{aligned}{}[e_3,e_2,e_3]_{\mathcal {L}}=-e_2,\quad [e_3,e_3,e_2]_{\mathcal {L}}=e_2. \end{aligned}$$

For a matrix $\left( \begin{array}{ccc} a_{11}&{}a_{12}&{}a_{13}\\ a_{21}&{}a_{22}&{}a_{23}\\ a_{31}&{}a_{32}&{}a_{33} \end{array}\right) ,$ define

$$\begin{aligned} Be_1=&a_{11}e_1+a_{21}e_2+a_{31}e_3,\quad Be_2=a_{12}e_1+a_{22}e_2+a_{32}e_3,\quad Be_3=a_{13}e_1+a_{23}e_2+a_{33}e_3. \end{aligned}$$

Then B is a Rota-Baxter operator if and only if

$$\begin{aligned} {[}Be_i,Be_j,Be_k]_{\mathcal {L}}=B([Be_i,Be_j,e_k]_{\mathcal {L}}+[Be_i,e_j,Be_k]_{\mathcal {L}}+[e_i,Be_j,Be_k]_{\mathcal {L}}),\quad i,j,k=1,2,3. \end{aligned}$$

• If $i=1,2, j=1, k=2,$ or $j=2, k=1$, we have

  $$\begin{aligned} a_{3i}a_{31}a_{12}=0,\quad a_{3i}a_{21}a_{32}=0,\quad a_{3i}a_{31}a_{32}=0. \end{aligned}$$

• If $i=1,2, j=1, k=3,$ or $j=3, k=1$, we have

  $$\begin{aligned} a_{3i}a_{21}a_{12}=0,\quad -a_{3i}a_{21}a_{33}+a_{3i}a_{31}a_{23}=-a_{3i}a_{21}a_{22},\quad a_{3i}a_{21}a_{32}=0. \end{aligned}$$

• If $i=1,2, j=2, k=3,$ or $j=3, k=2$, we have

  $$\begin{aligned} a_{3i}a_{12}(a_{33}+a_{22})=0,\quad a_{3i}a_{32}a_{23}=-a_{3i}a_{22}^{2},\quad a_{3i}a_{32}(a_{33}+a_{22})=0. \end{aligned}$$

  When $i=3$, consider the case of j and k as above respectively, we obtain

  $$\begin{aligned} \left\{ \begin{array}{rcl} a_{12}(a_{31}a_{22}-a_{21}a_{32}+a_{33}a_{31})&{}=&{}0,\\ a_{32}(a_{31}a_{22}-a_{21}a_{32}+a_{33}a_{31})&{}=&{}0,\\ a_{33}a_{21}a_{32}-a_{22}a_{21}a_{32}+a_{22}^{2}a_{31}&{}=&{}0,\\ \end{array}\right. \quad \quad \left\{ \begin{array}{rcl} a_{12}(a_{23}a_{32}-2a_{22}a_{33}-a_{33}^{2})&{}=&{}0,\\ a_{32}(a_{23}a_{32}-2a_{22}a_{33}-a_{33}^{2})&{}=&{}0,\\ a_{33}a_{23}a_{32}-a_{22}a_{23}a_{32}+2a_{22}^{2}a_{33}&{}=&{}0,\\ \end{array}\right. \end{aligned}$$
  $$\begin{aligned} {\left\{ \begin{array}{ll} a_{12}(a_{31}a_{23}-2a_{21}a_{33})=0,\\ a_{32}(a_{31}a_{23}-2a_{21}a_{33})=0,\\ a_{22}(a_{31}a_{23}-2a_{21}a_{33})=a_{33}(a_{31}a_{23}-a_{21}a_{33}). \end{array}\right. } \end{aligned}$$

Summarize the above discussions, we have

• (i) If $a_{12}=a_{21}=a_{22}=a_{31}=a_{32}=0, a_{33}\ne 0,$ then any $B=\left( \begin{array}{ccc}a_{11}&{}0&{}a_{13}\\ 0&{}0&{}a_{23}\\ 0&{}0&{}a_{33}\end{array}\right)$ is a Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$.

• (ii) If $a_{12}=a_{22}=a_{23}=a_{32}=a_{33}=0, a_{31}\ne 0,$ then any $B=\left( \begin{array}{ccc}a_{11}&{}0&{}a_{13}\\ a_{21}&{}0&{}0\\ a_{31}&{}0&{}0\end{array}\right)$ is a Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$.

• (iii) If $a_{12}=a_{22}=a_{32}=0, a_{31}\ne 0, a_{33}\ne 0,$ then any $B=\left( \begin{array}{ccc}a_{11}&{}0&{}a_{13}\\ \frac{a_{23}a_{31}}{a_{33}}&{}0&{}a_{23}\\ a_{31}&{}0&{}a_{33}\end{array}\right)$ is a Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$.

• (iv) If $a_{21}=a_{22}=a_{23}=a_{31}=a_{33}=0, a_{32}\ne 0,$ then any $B=\left( \begin{array}{ccc}a_{11}&{}a_{12}&{}a_{13}\\ 0&{}0&{}0\\ 0&{}a_{32}&{}0\end{array}\right)$ is a Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$.

• (v) If $a_{21}=a_{31}=0, a_{32}\ne 0, a_{33}\ne 0,$ then any $B=\left( \begin{array}{ccc}a_{11}&{}a_{12}&{}a_{13}\\ 0&{}-a_{33}&{}-\frac{a_{33}^{2}}{a_{32}}\\ 0&{}a_{32}&{}a_{33}\end{array}\right)$ is a Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$.

• (vi) If $a_{31}=a_{32}=a_{33}=0,$ then any $B=\left( \begin{array}{ccc} a_{11}&{}a_{12}&{}a_{13}\\ a_{21}&{}a_{22}&{}a_{23}\\ 0&{}0&{}0 \end{array}\right)$ is a Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$.

Now we give the following characterization of a relative Rota-Baxter operator in terms of graphs.

Proposition 2.12

A linear map $B:V\rightarrow \mathcal {L}$ is a relative Rota-Baxter operator on a 3-Leibniz algebra $\mathcal {L}$ with respect to a representation (V; l, m, r) if and only if the graph $Gr(B)=\{Bu+u|u\in V\}$ is a 3-Leibniz subalgebra of the semidirect product 3-Leibniz algebra $\mathcal {L}\ltimes _{l,m,r} V.$

Proof

Let B be a linear map. For all $Bu+u,Bv+v,Bw+w \in \mathcal {L}\oplus V$, we have

$$\begin{aligned}{}[Bu+u,Bv+v,Bw+w]_{\mathcal {L}\oplus V}=[Bu,Bv,Bw]_{\mathcal {L}}+l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u. \end{aligned}$$

Therefore, the graph $Gr(B)=\{Bu+u|u\in V\}$ is a subalgebra of the semidirect product 3-Leibniz algebra $\mathcal {L}\ltimes _{l,m,r} V$ if and only if B satisfies (16), which implies that B is a relative Rota-Baxter operator on a 3-Leibniz algebra $\mathcal {L}$ with respect to a representation (V; l, m, r). $\square$

Since the graph Gr(B) is isomorphic to V as vector spaces, so there is an induced 3-Leibniz algebra structure on V.

Proposition 2.13

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). Then $(V, [\cdot ,\cdot ,\cdot ]_B)$ is a 3-Leibniz algebra, called the descendent 3-Leibniz algebra of B, where $[\cdot ,\cdot ,\cdot ]_{B}:\otimes ^{3}V\rightarrow V$ is defined by

$$\begin{aligned} {[}u,v,w]_{B}=l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u,\quad \forall u,v,w \in V. \end{aligned}$$

(17)

Moreover, B is a homomorphism from the 3-Leibniz algebra $(V,[\cdot ,\cdot ,\cdot ]_{B})$ to the initial 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$.

Proof

By (16), B is a homomorphism from the 3-Leibniz algebra $(V,[\cdot ,\cdot ,\cdot ]_{B})$ to the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$. $\square$

The following results illustrate that 3-pre-Leibniz algebras can be viewd as the underlying algebraic structures of relative Rota-Baxter operators on 3-Leibniz algebras.

Theorem 2.14

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). Then $(V, [\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}, [\![ {\cdot ,\cdot ,\cdot }]\!] )$ is a 3-pre-Leibniz algebra whose sub-adjacent 3-Leibniz algebra is exactly the above descendent 3-Leibniz algebra, where

$$\begin{aligned}{}[u,v,w]=l(Bu,Bv)w,\quad \{u,v,w\}=m(Bu,Bw)v,\quad [\![ {u,v,w}]\!] =r(Bv,Bw)u, \quad \forall u,v,w \in V. \end{aligned}$$

(18)

Proof

For all $u_1,u_2,u_3,u_4,u_5\in V$, by (2), (13) and (16), we have

$$\begin{aligned}{} & {} [u_1,u_2,[u_3,u_4,u_5]]-[[u_1,u_2,u_3]_A,u_4,u_5]-[u_3,[u_1,u_2,u_4]_A,u_5]-[u_3,u_4,[u_1,u_2,u_5]]\\= & {} l(Bu_1,Bu_2)l(Bu_3,Bu_4)u_5-l\Big (B(l(Bu_1,Bu_2)u_3+m(Bu_1,Bu_3)u_2+r(Bu_2,Bu_3)u_1),Bu_4\Big )u_5\\{} & {} -l\Big (Bu_3,B(l(Bu_1,Bu_2)u_4+m(Bu_1,Bu_4)u_2+r(Bu_2,Bu_4)u_1)\Big )u_5-l(Bu_3,Bu_4)l(Bu_1,Bu_2)u_5\\= & {} l(Bu_1,Bu_2)l(Bu_3,Bu_4)u_5-l([Bu_1,Bu_2,Bu_3]_{\mathcal {L}},Bu_4)u_5\\{} & {} -l(Bu_3,[Bu_1,Bu_2,Bu_4]_{\mathcal {L}})u_5-l(Bu_3,Bu_4)l(Bu_1,Bu_2)u_5\\= & {} 0, \end{aligned}$$

which implies that (8) in Definition 2.5 holds. Similarly, by (3)–(6), we can deduce that (9)–(12) hold respectively. Therefore, $(V, [\cdot ,\cdot ,\cdot ], \{\cdot ,\cdot ,\cdot \}, [\![ {\cdot ,\cdot ,\cdot }]\!] )$ is a 3-pre-Leibniz algebra. By Theorem 2.7, we deduce that $(V,[\cdot ,\cdot ,\cdot ]_A)$ is a 3-Leibniz algebra. Obviously, its sub-adjacent 3-Leibniz algebra is exactly the 3-Leibniz algebra given in Proposition 2.13. Then the result follows. $\square$

Next we give a necessary and sufficient condition for the existence of a compatible 3-pre-Leibniz algebra structure on a 3-Leibniz algebra.

Proposition 2.15

Let $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ be a 3-Leibniz algebra. There exists a compatible 3-pre-Leibniz algebra if and only if there exists an invertible relative Rota-Baxter operator $B:V\rightarrow \mathcal {L}$ on $\mathcal {L}$ with respect to a representation (V; l, m, r).

Proof

The necessary condition follows from Proposition 2.10 directly. Next we prove the sufficient condition.

Let $B:V\rightarrow \mathcal {L}$ be an invertible relative Rota-Baxter operator on $\mathcal {L}$ with respect to a representation (V; l, m, r). By Theorem 2.14, there is a 3-pre-Leibniz algebra on V given by

$$\begin{aligned} {[}u,v,w]=l(Bu,Bv)w,\quad \{u,v,w\}=m(Bu,Bw)v,\quad [\![ {u,v,w}]\!] =r(Bv,Bw)u, \quad \forall u,v,w \in V. \end{aligned}$$

Since B is an invertible relative Rota-Baxter operator, we obtain that

$$\begin{aligned} {[}x,y,z]= & {} B[B^{-1}x,B^{-1}y,B^{-1}z]=B(l(x,y)B^{-1}z),\\ \{x,y,z\}= & {} B\{B^{-1}x,B^{-1}y,B^{-1}z\}=B(m(x,z)B^{-1}y),\\ [\![ {x,y,z}]\!]= & {} B[\![ {B^{-1}x,B^{-1}y,B^{-1}z}]\!] =B(r(y,z)B^{-1}x), \end{aligned}$$

defines a 3-pre-Leibniz algebra structures on $\mathcal {L}$. By (16), we have

$$\begin{aligned}{} & {} {[}x,y,z]+\{x,y,z\}+[\![ {x,y,z}]\!] \\{} & {} \quad =B(l(x,y)B^{-1}z)+B(m(x,z)B^{-1}y)+B(r(y,z)B^{-1}x)\\{} & {} \quad =B(l(B(B^{-1}x,B^{-1}y))B^{-1}z)+B(m(B(B^{-1}x,B^{-1}z))B^{-1}y)+B(r(B(B^{-1}y,B^{-1}z))B^{-1}x)\\{} & {} \quad =[x,y,z]_\mathcal {L}. \end{aligned}$$

The proof is finished. $\square$

3 Maurer-Cartan Characterizations of Relative Rota-Baxter Operators

In this section, we construct a Lie 3-algebra whose Maurer-Cartan elements are relative Rota-Baxter operators  on 3-Leibniz algebras. Moreover, we obtain the twisted $L_{\infty }$-algebra that controls deformations of relative Rota-Baxter operators  on 3-Leibniz algebras.

3.1 $L_\infty$-Algebras and Derived Brackets

A permutation $\sigma \in {\mathbb {S}}_n$ is called an $(i,n-i)$-shuffle if $\sigma (1)<\cdots <\sigma (i)$ and $\sigma (i+1)<\cdots <\sigma (n)$. If $i=0$ or n we assume $\sigma =\textrm{Id}$. The set of all $(i,n-i)$-shuffles will be denoted by ${\mathbb {S}}_{(i,n-i)}$.

Definition 3.1

([31]) An $L_\infty$-algebra is a ${\mathbb {Z}}$-graded vector space ${\mathfrak {g}}=\oplus _{k\in {\mathbb {Z}}}{\mathfrak {g}}^k$ equipped with a collection $(k\ge 1)$ of linear maps $l_k:\otimes ^k{\mathfrak {g}}\rightarrow {\mathfrak {g}}$ of degree 1 with the property that, for any homogeneous elements $x_1,\cdots ,x_n\in {\mathfrak {g}}$, we have

• (i) (graded symmetry) for every $\sigma \in {\mathbb {S}}_{n}$,

  $$\begin{aligned} l_n(x_{\sigma (1)},\cdots ,x_{\sigma (n-1)},x_{\sigma (n)})=\varepsilon (\sigma )l_n(x_1,\cdots ,x_{n-1},x_n). \end{aligned}$$

• (ii) (generalized Jacobi identity) for all $n\ge 1$,

  $$\begin{aligned} \sum _{i=1}^{n}\sum _{\sigma \in {\mathbb {S}}_{(i,n-i)} }\varepsilon (\sigma )l_{n-i+1}(l_i(x_{\sigma (1)},\cdots ,x_{\sigma (i)}),x_{\sigma (i+1)},\cdots ,x_{\sigma (n)})=0. \end{aligned}$$

The notion of a Lie k-algebra was introduced in [24], see [16] for more applications. Lie k-algebra is special $L_\infty$-algebra, in which only the k-ary bracket is nonzero.

Definition 3.2

A Lie 3-algebra is a ${\mathbb {Z}}$-graded vector space ${\mathfrak {g}}=\oplus _{k\in {\mathbb {Z}}}{\mathfrak {g}}^k$ equipped with a trilinear bracket $\{\cdot ,\cdot ,\cdot \}_{\mathfrak {g}}:{\mathfrak {g}}\otimes {\mathfrak {g}}\otimes {\mathfrak {g}}\,\rightarrow \,{\mathfrak {g}}$ of degree 1, satisfying

• (i) (graded symmetry) for all homogeneous elements $x_1,x_2,x_3\in {\mathfrak {g}}$,

  $$\begin{aligned} \{x_1,x_2,x_3\}_{\mathfrak {g}}=(-1)^{x_1x_2}\{x_2,x_1,x_3\}_{\mathfrak {g}}=(-1)^{x_2x_3}\{x_1,x_3,x_2\}_{\mathfrak {g}}. \end{aligned}$$

  (19)

• (ii) (generalized Jacobi identity) for all homogeneous elements $x_{i}\in {\mathfrak {g}}, 1\le i\le 5$,

  $$\begin{aligned} \sum _{\sigma \in {\mathbb {S}}_{5} }\varepsilon (\sigma )\{\{x_{\sigma (1)},x_{\sigma (2)},x_{\sigma (3)}\}_{\mathfrak {g}},x_{\sigma (4)},x_{\sigma (5)}\}_{\mathfrak {g}}=0. \end{aligned}$$

  (20)

Definition 3.3

A Maurer-Cartan element of an $L_\infty$-algebra $({\mathfrak {g}},\{l_i\}_{i=1}^{+\infty })$ is an element $\alpha \in {\mathfrak {g}}^0$ satisfying the Maurer-Cartan equation $\sum _{n=1}^{+\infty } \frac{1}{n!}l_n(\alpha ,\cdots ,\alpha )=0.$ Specially, $\alpha \in {\mathfrak {g}}^0$ is a Maurer-Cartan element of a Lie 3-algebra $({\mathfrak {g}},\{\cdot ,\cdot ,\cdot \}_{\mathfrak {g}})$ if and only if $\frac{1}{3!}\{\alpha ,\alpha ,\alpha \}_{\mathfrak {g}}=0.$

Let $\alpha$ be a Maurer-Cartan element of a Lie 3-algebra $({\mathfrak {g}},\{\cdot ,\cdot ,\cdot \}_{\mathfrak {g}})$. Define $l_k^{\alpha }:\otimes ^k{\mathfrak {g}}\rightarrow {\mathfrak {g}}$ by

$$\begin{aligned} l_k^{\alpha }(x_1,\cdots ,x_k)= & {} \sum \limits _{n=0}^{+\infty }\frac{1}{n!}l_{k+n}\{\underbrace{\alpha ,\cdots ,\alpha }_{n},x_1,\cdots ,x_k\},\quad k\ge 1. \end{aligned}$$

(21)

Lemma 3.4

([22]) With the above notations, $({\mathfrak {g}},\{l_k^{\alpha }\}_{k=1}^{+\infty })$ is an $L_\infty$-algebra. Moreover, $\alpha +\alpha '$ is a Maurer-Cartan element of $({\mathfrak {g}},\{l_k\}_{k=1}^{+\infty })$ if and only if $\alpha '$ is a Maurer-Cartan element of the twisted $L_\infty$-algebra $({\mathfrak {g}},\{l_k^{\alpha }\}_{k=1}^{+\infty })$.

In the sequel, we recall Th. Voronov’s derived brackets [36], which is a useful tool to construct explicit $L_\infty$-algebras.

Definition 3.5

([36]) A V-data consists of a quadruple $(F,{\mathfrak {h}},\mathcal {P},\Delta )$ where

• $(F,[\cdot ,\cdot ])$ is a graded Lie algebra;

• ${\mathfrak {h}}$ is an abelian graded Lie subalgebra of $(F,[\cdot ,\cdot ])$;

• $\mathcal {P}:F\,\rightarrow \,F$ is a projection, that is $\mathcal {P}\circ \mathcal {P}=\mathcal {P}$, whose image is ${\mathfrak {h}}$ and kernel is a graded Lie subalgebra of $(F,[\cdot ,\cdot ])$;

• $\Delta$ is an element in $\ker (\mathcal {P})^1$ such that $[\Delta ,\Delta ]=0$.

Theorem 3.6

([36]) Let $(F,{\mathfrak {h}},\mathcal {P},\Delta )$ be a V-data. Then $({\mathfrak {h}},\{{l_k}\}_{k=1}^{+\infty })$ is an $L_\infty$-algebra where

$$\begin{aligned} l_k(a_1,\cdots ,a_k)=\mathcal {P}\underbrace{[\cdots [[}_k\Delta ,a_1],a_2],\cdots ,a_k],\quad \text{ for } \text{ homogeneous }~ a_1,\cdots ,a_k\in {\mathfrak {h}}. \end{aligned}$$

(22)

We call $\{{l_k}\}_{k=1}^{+\infty }$ the higher derived brackets of the V-data $(F,{\mathfrak {h}},\mathcal {P},\Delta )$.

3.2 The Controlling Algebra of relative Rota-Baxter operators

Let $\mathcal {L}$ be vector spaces. Consider the graded vector space

$$\begin{aligned} C_{\textsf {3Leib}}^{*}(\mathcal {L},\mathcal {L})=& {}\oplus _{p\ge 0}C_{\textsf {3Leib}}^{p}(\mathcal {L},\mathcal {L})\\=& {}\oplus _{p\ge 0}\textrm{Hom}(\underbrace{(\otimes ^{2}\mathcal {L})\otimes \cdots \otimes (\otimes ^{2}\mathcal {L})}_{p}\otimes \mathcal {L},\mathcal {L}). \end{aligned}$$

Theorem 3.7

([30]) The graded vector space $C_{\textsf {3Leib}}^{*}(\mathcal {L},\mathcal {L})$ equipped with the graded bracket

$$\begin{aligned}{}[P,Q]_{\textsf {3Leib}}=P\circ Q-(-1)^{pq}Q\circ P, \quad \forall P \in C_{\textsf {3Leib}}^{p}(\mathcal {L},\mathcal {L}), Q\in C_{\textsf {3Leib}}^{q}(\mathcal {L},\mathcal {L}), \end{aligned}$$

is a graded Lie algebra, where $P\circ Q\in C_{\textsf {3Leib}}^{p+q}(\mathcal {L},\mathcal {L})$ is defined by

$$\begin{aligned}{} & {} (P\circ Q)(\mathfrak {X}_{1},\cdots ,\mathfrak {X}_{p+q},x)\\= & {} \sum _{k=1}^{p}(-1)^{(k-1)q}\sum _{\sigma \in S(k-1,q)}(-1)^{\sigma }P\big (\mathfrak {X}_{\sigma (1)},\cdots ,\mathfrak {X}_{\sigma (k-1)},Q(\mathfrak {X}_{\sigma (k)},\cdots ,\mathfrak {X}_{\sigma (k+q-1)},x_{k+q})\otimes y_{k+q},\\{} & {} \mathfrak {X}_{k+q+1},\cdots ,\mathfrak {X}_{p+q},x\big )\\{} & {} +\sum _{k=1}^{p}(-1)^{(k-1)q}\sum _{\sigma \in S(k-1,q)}(-1)^{\sigma }P\big (\mathfrak {X}_{\sigma (1)},\cdots ,\mathfrak {X}_{\sigma (k-1)},x_{k+q}\otimes Q(\mathfrak {X}_{\sigma (k)},\cdots ,\mathfrak {X}_{\sigma (k+q-1)},y_{k+q}),\\{} & {} \mathfrak {X}_{k+q+1},\cdots ,\mathfrak {X}_{p+q},x\big )\\{} & {} +\sum _{\sigma \in S(p,q)}(-1)^{pq}(-1)^{\sigma }P\big (\mathfrak {X}_{\sigma (1)},\cdots ,\mathfrak {X}_{\sigma (p)},Q(\mathfrak {X}_{\sigma (p+1)},\cdots ,\mathfrak {X}_{\sigma (p+q)},x)\big ), \end{aligned}$$

for all $\mathfrak {X}_{i}=x_{i}\otimes y_{i}\in \otimes ^{2}\mathcal {L}$, $~i=1,2,\cdots ,p+q$ and $x\in \mathcal {L}.$

Remark 3.8

For $\mu \in \textrm{Hom}(\otimes ^{3}\mathcal {L},\mathcal {L})$, we have

$$\begin{aligned} {[}\mu ,\mu ]_{\textsf {3Leib}}(\mathfrak {X}_{1},\mathfrak {X}_{2},x)= & {} 2(\mu \circ \mu )(\mathfrak {X}_{1},\mathfrak {X}_{2},x)\\= & {} 2\big (\mu (\mu (x_{1},y_{1},x_{2}),y_{2},x)+\mu (x_{2},\mu (x_{1},y_{1},y_{2}),x)\\{} & {} -\mu (x_{1},y_{1},\mu (x_{2},y_{2},x))+\mu (x_{2},y_{2},\mu (x_{1},y_{1},x))\big ). \end{aligned}$$

Thus, $\mu$ defines a 3-Leibniz algebra structure on $\mathcal {L}$ if and only if $[\mu , \mu ]_{\textsf {3Leib}}=0$, i.e. $\mu$ is a Maurer-Cartan element of the graded Lie algebra $\big (C_{\textsf {3Leib}}^{*}(\mathcal {L}, \mathcal {L}),[\cdot ,\cdot ]_{\textsf {3Leib}}\big )$.

If $\mathcal {L} = \mathcal {L}_1 \oplus \mathcal {L}_2$, the direct sum of two vector spaces $\mathcal {L}_1$ and $\mathcal {L}_2$, then the space $C_{\textsf {3Leib}}^{p}(\mathcal {L}, \mathcal {L})$ is quite complicated. To describe it more precisely, we denote by $\mathcal {L}^{m,n}$ the subspace of $\otimes ^{p}(\otimes ^{2}(\mathcal {L}_1\oplus \mathcal {L}_2))\otimes (\mathcal {L}_1\oplus \mathcal {L}_2)$ which contains the numbers of $\mathcal {L}_1$ and $\mathcal {L}_2$ are $m$ and $n$, respectively. Then the vector space $\otimes ^{p}(\otimes ^{2}(\mathcal {L}_1\oplus \mathcal {L}_2))\otimes (\mathcal {L}_1\oplus \mathcal {L}_2)$ is isomorphic to the direct sum of $\mathcal {L}^{m,n},m+n=2p+1$. Furthermore, we have the following isomorphism.

$$\begin{aligned} C_{\textsf {3Leib}}^{p}(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2)\cong \sum \limits _{m+n=2p+1}\textrm{Hom}(\mathcal {L}^{m,n},\mathcal {L}_1)\oplus \sum \limits _{m+n=2p+1}\textrm{Hom}(\mathcal {L}^{m,n},\mathcal {L}_2). \end{aligned}$$

(23)

An element $f\in \textrm{Hom}(\mathcal {L}^{m,n},\mathcal {L}_1)$ (resp. $f\in \textrm{Hom}(\mathcal {L}^{m,n},\mathcal {L}_2)$) naturally gives an element $\hat{f}\in C_{\textsf {3Leib}}^{p}(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2)$, which is called its lift. For example, the lifts of linear maps $\mu :\otimes ^{3}\mathcal {L}_1 \rightarrow \mathcal {L}_1$ and $\tau :\otimes ^2 \mathcal {L}_1 \otimes \mathcal {L}_2 \rightarrow \mathcal {L}_2$ are defined by

$$\begin{aligned} \hat{\mu }((x_1,u_1),(x_2,u_2),(x_3,u_3))= & {} (\mu (x_1,x_2,x_3),0),\\ \hat{\tau }((x_1,u_1),(x_2,u_2),(x_3,u_3))= & {} (0,\tau (x_1,x_2)(u_3)). \end{aligned}$$

Let (V; l, m, r) be a representation of a 3-Leibniz algebra $(\mathcal {L}, [\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$. Usually we use $\mu$ to indicate the 3-Leibniz bracket $[\cdot ,\cdot ,\cdot ]_{\mathcal {L}}$. In the sequel, we will view l, m, r as elements in $\textrm{Hom}(\otimes ^2\mathcal {L}\otimes V,V)$, $\textrm{Hom}(\mathcal {L}\otimes V\otimes \mathcal {L},V)$ and $\textrm{Hom}(V\otimes (\otimes ^2\mathcal {L}),V)$ respectively. Then the semidirect product 3-Leibniz algebra given by Lemma 2.4 corresponds to

$$\begin{aligned}{} & {} (\hat{\mu }+\hat{l}+\hat{m}+\hat{r})\Big ((x_1,u_1),(x_2,u_2),(x_3,u_3)\Big )\nonumber \\= & {} \Big (\mu (x_1,x_2,x_3),l(x_1,x_2)u_3+m(x_1,x_3)u_2+r(x_2,x_3)u_1\Big ). \end{aligned}$$

(24)

By Theorem 3.7, we have

$$\begin{aligned} {[}\hat{\mu }+\hat{l}+\hat{m}+\hat{r},\hat{\mu }+\hat{l}+\hat{m}+\hat{r}]_{\textsf {3Leib}}=0. \end{aligned}$$

Lemma 3.9

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). For all $x,y,z \in \mathcal {L}, u,v,w \in V$, we have

$$\begin{aligned}{} & {} {[}{[}\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}(x+u,y+v,z+w)\\= & {} 2\Big ([Bu,Bv,z]_{\mathcal {L}}+[Bu,y,Bw]_{\mathcal {L}}+[x,Bv,Bw]_{\mathcal {L}}+l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u\\{} & {} -B\big (l(Bu,y)w+m(Bu,z)v+l(x,Bv)w+r(Bv,z)u+m(x,Bw)v+r(y,Bw)u\big )\Big ). \end{aligned}$$

Proof

It follows from straightforward computations. $\square$

Proposition 3.10

Let (V; l, m, r) be a representation of a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$. Then we have a V-data $(F,{\mathfrak {h}},\mathcal {P},\Delta )$ as follows:

• the graded Lie algebra $(F,[\cdot ,\cdot ])$ is given by $(C^*_{\textsf {3Leib}}(\mathcal {L}\oplus V,\mathcal {L}\oplus V),[\cdot ,\cdot ]_{\textsf {3Leib}})$;

• the abelian graded Lie subalgebra ${\mathfrak {h}}$ is given by

  $$\begin{aligned} {\mathfrak {h}}=C_{\textsf {3Leib}}^*(V,\mathcal {L})=\oplus _{n=0}^{+\infty }C_{\textsf {3Leib}}^{n}(V,\mathcal {L})=\oplus _{n\ge 0}\textrm{Hom}(\underbrace{(\otimes ^{2} V)\otimes \cdots \otimes (\otimes ^{2}V)}_{n}\otimes V, \mathcal {L}); \end{aligned}$$

• $\mathcal {P}:F\,\rightarrow \,F$ is the projection onto the subspace ${\mathfrak {h}}$;

• $\Delta =\hat{\mu }+\hat{l}+\hat{m}+\hat{r}$.

Consequently, we obtain a Lie 3-algebra $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_3)$, where

$$\begin{aligned} l_3(P,Q,R)= & {} [[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},P]_{\textsf {3Leib}},Q]_{\textsf {3Leib}},R]_{\textsf {3Leib}}, \end{aligned}$$

for all $P \in C_{\textsf {3Leib}}^{p}(V,\mathcal {L}), Q \in C_{\textsf {3Leib}}^{q}(V,\mathcal {L})$ and $R\in C_{\textsf {3Leib}}^{r}(V,\mathcal {L})$.

Proof

By Theorem 3.6, $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),\{l_k\}_{k=1}^{+\infty })$ is an $L_\infty$-algebra, where $l_k$ is given by (22). For all $P \in C_{\textsf {3Leib}}^{p}(V,\mathcal {L}), Q \in C_{\textsf {3Leib}}^{q}(V,\mathcal {L})$ and $R\in C_{\textsf {3Leib}}^{r}(V,\mathcal {L})$, by Lemma 3.9, we obtain

$$\begin{aligned}{} & {} [\hat{\mu }+\hat{l}+\hat{m}+\hat{r},P]_{\textsf {3Leib}}\in \ker (\mathcal {P}),\\{} & {} [[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},P]_{\textsf {3Leib}},Q]_{\textsf {3Leib}}\in \ker (\mathcal {P}), \end{aligned}$$

which implies that $l_1=0$ and $l_2=0$. Similarly, we can deduce that $l_k=0,$ $k\ge 4$. Therefore, $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_3)$ is a Lie 3-algebra. $\square$

Theorem 3.11

Let (V; l, m, r) be a representation of a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$. Then Maurer-Cartan elements of the Lie 3-algebra $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_3)$ are precisely relative Rota-Baxter operators on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to the representation (V; l, m, r).

Proof

Let B be a Maurer-Cartan element of the Lie 3-algebra $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_3)$. We have

$$\begin{aligned}\frac{1}{3!}l_3(B,B,B)(u,v,w){} & {} =\frac{1}{3!}[[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}(u,v,w)\\{} & {} \quad =[Bu,Bv,Bw]_{\mathcal {L}}-B(l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u)\\{} & {} \quad =0, \end{aligned}$$

which indicates that B is a relative Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to the representation (V; l, m, r). $\square$

Proposition 3.12

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). Then $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_1^{B},l_2^{B},l_3^{B})$ is a twisted $L_{\infty }$-algebra, where $l_1^{B},l_2^{B},l_3^{B}$ are given by

$$\begin{aligned} l_1^{B}(P)= & {} \frac{1}{2}l_3(B,B,P), \end{aligned}$$

(25)

$$\begin{aligned} l_2^{B}(P,Q)= & {} l_3(B,P,Q),\end{aligned}$$

(26)

$$\begin{aligned} l_3^{B}(P,Q,R)= & {} l_3(P,Q,R),\end{aligned}$$

(27)

$$\begin{aligned} l_k^{B}= & {} 0,\quad k\ge 4, \end{aligned}$$

(28)

for all $P\in C_{\textsf {3Leib}}^p(V,\mathcal {L}),Q\in C_{\textsf {3Leib}}^q(V,\mathcal {L})$ and $R\in C_{\textsf {3Leib}}^r(V,\mathcal {L})$.

Proof

Since the relative Rota-Baxter operator B is a Maurer-Cartan element of the Lie 3-algebra $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_3),$ by Lemma 3.4, we have the conclusions. $\square$

Actually, the above twisted $L_{\infty }$-algebra controls deformations of relative Rota-Baxter operators on 3-Leibniz algebras.

Theorem 3.13

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). Then for a linear map $B':V\rightarrow \mathcal {L}$, $B+B'$ is also a relative Rota-Baxter operator if and only if $B'$ is a Maurer-Cartan element of the twisted $L_\infty$-algebra $\big (C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_1^{B},l_2^{B},l_3^{B}\big )$, that is, $B'$ satisfies the Maurer-Cartan equation:

$$\begin{aligned} l_1^{B}(B')+\frac{1}{2}l_2^{B}(B',B')+\frac{1}{3!}l_3^{B}(B',B',B')=0. \end{aligned}$$

Proof

By Theorem 3.11, $B+B'$ is a relative Rota-Baxter operator if and only if

$$\begin{aligned} \frac{1}{3!}l_3(B+B',B+B',B+B')=0. \end{aligned}$$

By $l_3(B,B,B)=0,$ the above equality is equivalent to

$$\begin{aligned} \frac{1}{2}l_3(B,B,B')+\frac{1}{2}l_3(B,B',B')+\frac{1}{6}l_3(B',B',B')=0, \end{aligned}$$

which implies that $l_1^{B}(B')+\frac{1}{2}l_2^{B}(B',B')+\frac{1}{3!}l_3^{B}(B',B',B')=0.$

Therefore $B'$ is a Maurer-Cartan element of the twisted $L_\infty$-algebra $\big (C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_1^{B},l_2^{B},l_3^{B}\big )$. $\square$

4 The Cohomology and Infinitesimal Deformations of relative Rota-Baxter operators

In this section, we construct a representation of the 3-Leibniz algebra $(V, [\cdot ,\cdot ,\cdot ]_B)$ on the vector space $\mathcal {L},$ and define the cohomology of relative Rota-Baxter operators on 3-Leibniz algebras. Next, we study infinitesimal deformations of relative Rota-Baxter operators using the second cohomology group.

4.1 The Cohomology of Relative Rota-Baxter Operators on 3-Leibniz Algebras

First, we recall the cohomology theory of 3-Leibniz algebras.

Let (V; l, m, r) be a representation of a 3-Leibniz algebra $(\mathcal {L}, [\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$. Denote by ${\mathfrak {C}}_{\textsf {3Leib}}^{n}(\mathcal {L};V)$ the set of n-cochains:

$$\begin{aligned} \mathfrak {\mathfrak {C}}_{\textsf {3Leib}}^{n}(\mathcal {L};V):=\{\text{ linear } \text{ maps }~ f:\underbrace{(\otimes ^{2}\mathcal {L})\otimes \cdots \otimes (\otimes ^{2}\mathcal {L})}_{n-1}\otimes \mathcal {L}\rightarrow V\},\quad n\ge 1. \end{aligned}$$

The coboundary operator $\partial :{\mathfrak {C}}_{\textsf {3Leib}}^{n}(\mathcal {L};V)\rightarrow {\mathfrak {C}}_{\textsf {3Leib}}^{n+1}(\mathcal {L};V)$ is given by

$$\begin{aligned}{} & {} (\partial f)(X_{1},X_{2},\cdots ,X_{n},z)\\{} & {} \quad =\sum _{1\le j<k\le n}(-1)^{j}f(X_{1},\cdots ,\hat{X_{j}},\cdots ,X_{k-1},[x_{j},y_{j},x_{k}]_{\mathcal {L}}\otimes y_{k},X_{k+1},\cdots ,X_{n},z)\\{} & {} \qquad +\sum _{1\le j<k\le n}(-1)^{j}f(X_{1},\cdots ,\hat{X_{j}},\cdots ,X_{k-1},x_{k}\otimes [x_{j},y_{j},y_{k}]_{\mathcal {L}},X_{k+1},\cdots ,X_{n},z)\\{} & {} \qquad +\sum _{j=1}^{n}\Big ((-1)^{j}f(X_{1},\cdots ,\hat{X_{j}},\cdots ,X_{n},[X_{j},z]_{\mathcal {L}})+(-1)^{j+1}l(X_{j})f(X_{1},\cdots ,\hat{X_{j}},\cdots ,X_{n},z)\Big )\\{} & {} \qquad +(-1)^{n+1}\Big (m(x_{n},z)f(X_{1},\cdots ,X_{n-1},y_{n})+r(y_{n},z)f(X_{1},\cdots ,X_{n-1},x_{n})\Big ), \end{aligned}$$

where $X_{i}=x_{i}\otimes y_{i}\in \otimes ^{2}\mathcal {L},~i=1, 2, \cdots , n$ and $z\in \mathcal {L}$. It was proved in [13, 32] that $\partial \circ \partial =0.$ Thus, $(\oplus _{n=1}^{+\infty }{\mathfrak {C}}_{\textsf {3Leib}}^{n}(\mathcal {L};V),\partial )$ is a cochain complex.

Definition 4.1

The cohomology of the 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ with coefficients in V is the cohomology of the cochain complex $(\oplus _{n=1}^{+\infty }{\mathfrak {C}}_{\textsf {3Leib}}^{n}(\mathcal {L};V),\partial )$. Denote by $Z^n_{\textsf {3Leib}}(\mathcal {L};V)$ and $B^n_{\textsf {3Leib}}(\mathcal {L};V)$ the set of n-cocycles and the set of n-coboundaries, respectively. The n-th cohomology group is defined by

$$\begin{aligned} H^n_{\textsf {3Leib}}(\mathcal {L};V)=Z^n_{\textsf {3Leib}}(\mathcal {L};V)/B^n_{\textsf {3Leib}}(\mathcal {L};V). \end{aligned}$$

(29)

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). By Proposition 2.13, there is a 3-Leibniz algebra $(V,[\cdot ,\cdot ,\cdot ]_{B})$. Define linear maps $l_{B}, m_{B}, r_{B}:\otimes ^{2}V\rightarrow \mathfrak {gl}(\mathcal {L})$ by

$$\begin{aligned} l_{B}(u,v)(x):= & {} [Bu, Bv, x]_{\mathcal {L}}-B\big (m(Bu,x)v+r(Bv,x)u\big ), \end{aligned}$$

(30)

$$\begin{aligned} m_{B}(u,v)(x):= & {} [Bu, x, Bv]_{\mathcal {L}}-B\big (l(Bu,x)v+r(x,Bv)u\big ),\end{aligned}$$

(31)

$$\begin{aligned} r_{B}(u,v)(x):= & {} [x, Bu, Bv]_{\mathcal {L}}-B\big (l(x,Bu)v+m(x,Bv)u\big ). \end{aligned}$$

(32)

Proposition 4.2

With the above notations, $(\mathcal {L}; l_{B},m_{B},r_{B})$ is a representation of the 3-Leibniz algebra $(V, [\cdot ,\cdot ,\cdot ]_B)$.

Proof

For all $x \in \mathcal {L}, u_i \in V,~i=1,\cdots ,4$, by (1), (16) and (17), we have

$$\begin{aligned}{} & {} \Big (l_{B}([u_1,u_2,u_3]_B,u_4)+l_{B}(u_3,[u_1,u_2,u_4]_B)+l_{B}(u_{3},u_{4})l_{B}(u_{1},u_{2})-l_{B}(u_{1},u_{2})l_{B}(u_{3},u_{4})\Big )(x)\\{} & {} \quad =[B[u_1,u_2,u_3]_B,Bu_4,x]_{\mathcal {L}}-B(m(B[u_1,u_2,u_3]_B,x)u_4)-B(r(Bu_4,x)([u_1,u_2,u_3]_B))\\{} & {} \qquad +[Bu_3,B[u_1,u_2,u_4]_B,x]_{\mathcal {L}}-B(m(Bu_3,x)([u_1,u_2,u_4]_B))-B(r(B[u_1,u_2,u_4]_B,x)u_3)\\{} & {} \qquad +l_{B}(u_{3},u_{4})\Big ([Bu_1,Bu_2,x]_{\mathcal {L}}-B(m(Bu_1,x)u_2)-B(r(Bu_2,x)u_1)\Big )\\{} & {} \qquad -l_{B}(u_{1},u_{2})\Big ([Bu_3,Bu_4,x]_{\mathcal {L}}-B(m(Bu_3,x)u_4)-B(r(Bu_4,x)u_3)\Big )\\{} & {} \quad =[[Bu_1,Bu_2,Bu_3]_{\mathcal {L}},Bu_4,x]_{\mathcal {L}}-B(m([Bu_1,Bu_2,Bu_3]_{\mathcal {L}},x)u_4)-B(r(Bu_4,x)l(Bu_1,Bu_2)u_3)\\{} & {} \qquad -B(r(Bu_4,x)m(Bu_1,Bu_3)u_2)-B(r(Bu_4,x)r(Bu_2,Bu_3)u_1)\\{} & {} \qquad +[Bu_3,[Bu_1,Bu_2,Bu_4]_{\mathcal {L}},x]_{\mathcal {L}}-B(m(Bu_3,x)l(Bu_1,Bu_2)u_4)-B(m(Bu_3,x)m(Bu_1,Bu_4)u_2)\\{} & {} \qquad -B(m(Bu_3,x)r(Bu_2,Bu_4)u_1)-B(r([Bu_1,Bu_2,Bu_4]_{\mathcal {L}},x)u_3)\\{} & {} \qquad +[Bu_3,Bu_4,[Bu_1,Bu_2,x]_{\mathcal {L}}]_{\mathcal {L}}-B(m(Bu_3,[Bu_1,Bu_2,x]_{\mathcal {L}})u_4)-B(r(Bu_4,[Bu_1,Bu_2,x]_{\mathcal {L}})u_3)\\{} & {} \qquad -[Bu_3,Bu_4,B(m(Bu_1,x)u_2)]_{\mathcal {L}}+B(m(Bu_3,B(m(Bu_1,x)u_2))u_4)+B(r(Bu_4,B(m(Bu_1,x)u_2))u_3)\\{} & {} \qquad -[Bu_3,Bu_4,B(r(Bu_2,x)u_1)]_{\mathcal {L}}+B(m(Bu_3,B(r(Bu_2,x)u_1))u_4)+B(r(Bu_4,B(r(Bu_2,x)u_1))u_3)\\{} & {} \qquad -[Bu_1,Bu_2,[Bu_3,Bu_4,x]_{\mathcal {L}}]_{\mathcal {L}}+B(m(Bu_1,[Bu_3,Bu_4,x]_{\mathcal {L}})u_2)+B(r(Bu_2,[Bu_3,Bu_4,x]_{\mathcal {L}})u_1)\\{} & {} \qquad +[Bu_1,Bu_2,B(m(Bu_3,x)u_4)]_{\mathcal {L}}-B(m(Bu_1,B(m(Bu_3,x)u_4))u_2)-B(r(Bu_2,B(m(Bu_3,x)u_4))u_1)\\{} & {} \qquad +[Bu_1,Bu_2,B(r(Bu_4,x)u_3)]_{\mathcal {L}}-B(m(Bu_1,B(r(Bu_4,x)u_3))u_2)-B(r(Bu_2,B(r(Bu_4,x)u_3))u_1)\\{} & {} \quad =-B(m([Bu_1,Bu_2,Bu_3]_{\mathcal {L}},x)u_4)-B(m(Bu_3,x)l(Bu_1,Bu_2)u_4)-B(m(Bu_3,[Bu_1,Bu_2,x]_{\mathcal {L}})u_4)\\{} & {} \qquad -B(r(Bu_4,x)l(Bu_1,Bu_2)u_3)-B(r([Bu_1,Bu_2,Bu_4]_{\mathcal {L}},x)u_3)-B(r(Bu_4,[Bu_1,Bu_2,x]_{\mathcal {L}})u_3)\\{} & {} \qquad -B(r(Bu_4,x)m(Bu_1,Bu_3)u_2)-B(m(Bu_3,x)m(Bu_1,Bu_4)u_2)-B(l(Bu_3,Bu_4)m(Bu_1,x)u_2)\\{} & {} \qquad -B(r(Bu_4,x)r(Bu_2,Bu_3)u_1)-B(m(Bu_3,x)r(Bu_2,Bu_4)u_1)-B(l(Bu_3,Bu_4)r(Bu_2,x)u_1)\\{} & {} \qquad +B(r(Bu_2,[Bu_3,Bu_4,x]_{\mathcal {L}})u_1)+B(m(Bu_1,[Bu_3,Bu_4,x]_{\mathcal {L}})u_2)\\{} & {} \qquad +B(l(Bu_1,Bu_2)r(Bu_4,x)u_3)+B(l(Bu_1,Bu_2)m(Bu_3,x)u_4)\\{} & {} \quad =0, \end{aligned}$$

which implies that (2) holds. Similarly, we can deduce that (3)–(6) hold. Therefore, $(\mathcal {L}; l_{B},m_{B},r_{B})$ is a representation of the 3-Leibniz algebra $(V, [\cdot ,\cdot ,\cdot ]_B)$. $\square$

Let $\partial _{B}:{\mathfrak {C}}_{\textsf {3Leib}}^{n}(V;\mathcal {L})\rightarrow {\mathfrak {C}}_{\textsf {3Leib}}^{n+1}(V;\mathcal {L}),~(n\ge 1)$ be the coboundary operator of the 3-Leibniz algebra $(V, [\cdot ,\cdot ,\cdot ]_{B})$ with coefficients in the representation $(\mathcal {L};l_{B},m_{B},r_{B})$. For all $\theta \in {\mathfrak {C}}_{\textsf {3Leib}}^{n}(V;\mathcal {L})$, $U_{i}=u_{i}\otimes v_{i}\in \otimes ^{2}V,~i=1,2,\ldots ,n$ and $u_{n+1} \in V$, we have

$$\begin{aligned}{} & {} \partial _{B}\theta (U_{1},U_{2},\ldots ,U_{n},u_{n+1})\\{} & {} \quad =\sum _{1\le j<k\le n}(-1)^{j}\theta \Big (U_{1},\ldots ,\hat{U_{j}},\ldots ,U_{k-1},[u_{j},v_{j},u_{k}]_{B}\otimes v_{k}+u_{k}\otimes [u_{j},v_{j},v_{k}]_{B},U_{k+1},\ldots ,U_{n},u_{n+1}\Big )\\{} & {} \qquad +\sum _{j=1}^{n}(-1)^{j}\theta \Big (U_{1},\ldots ,\hat{U_{j}},\ldots ,U_{n},[u_j,v_j,u_{n+1}]_{B}\Big )\\{} & {} \qquad +\sum _{j=1}^{n}(-1)^{j+1}\Big ([Bu_{j},Bv_{j},\theta (U_{1},\ldots ,\hat{U_{j}},\ldots ,U_{n},u_{n+1})]_{\mathcal {L}}\\{} & {} \qquad -B\big (m(Bu_{j},\theta (U_{1},\ldots ,\hat{U_{j}},\ldots ,U_{n},u_{n+1}))v_{j}\big )-B\big (r(Bv_{j},\theta (U_{1},\ldots ,\hat{U_{j}},\cdots ,U_{n},u_{n+1}))u_{j}\big )\Big )\\{} & {} \qquad +(-1)^{n+1}\Big ([Bu_{n},\theta (U_{1},U_{2},\ldots ,U_{n-1},v_{n}),Bu_{n+1}]_{\mathcal {L}}-B\big (l(Bu_{n},\theta (U_{1},\ldots ,U_{n-1},v_{n}))u_{n+1}\big )\\{} & {} \qquad -B\big (r(\theta (U_{1},\ldots ,U_{n-1},v_{n}),Bu_{n+1})u_{n})\Big )+(-1)^{n+1}\Big ([\theta (U_{1},U_{2},\ldots ,U_{n-1},u_{n}),Bv_{n},Bu_{n+1}]_{\mathcal {L}}\\{} & {} \qquad -B\big (l(\theta (U_{1},\ldots ,U_{n-1},u_{n}),Bv_{n})u_{n+1}\big )-B\big (m(\theta (U_{1},\ldots ,U_{n-1},u_{n}),Bu_{n+1})v_{n}\big )\Big ). \end{aligned}$$

It is obvious that $\theta \in \textrm{Hom}(V,\mathcal {L})$ is closed if and only if

$$\begin{aligned}{} & {} [Bu,Bv,\theta w]_{\mathcal {L}}+[\theta u,Bv,Bw]_{\mathcal {L}}+[Bu,\theta v,Bw]_{\mathcal {L}}-\theta \Big (l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u\Big )\\{} & {} \quad =B\Big (l(\theta u,Bv)w+l(Bu,\theta v)w+m(\theta u,Bw)v+m(Bu,\theta w)v+r(\theta v,Bw)u+r(Bv,\theta w)u\Big ). \end{aligned}$$

For any $\mathfrak {X} \in \otimes ^2\mathcal {L}$, we define $\delta (\mathfrak {X}):V\rightarrow \mathcal {L}$ by

$$\begin{aligned} \delta (\mathfrak {X})v=Bl(\mathfrak {X})v-[\mathfrak {X},Bv]_{\mathcal {L}},\quad \forall v \in V. \end{aligned}$$

(33)

Proposition 4.3

Let B be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). Then $\delta (\mathfrak {X})$ is a 1-cocycle on the 3-Leibniz algebra $(V,[\cdot ,\cdot ,\cdot ]_{B})$ with coefficients in $(\mathcal {L};l_{B},m_{B},r_{B})$.

Proof

For all $u,v,w \in V$, by (16) and (17), we have

$$\begin{aligned} (\partial _{B}\delta (\mathfrak {X}))(u,v,w){} & {} =-Bl(\mathfrak {X})([u,v,w]_{B})+[\mathfrak {X},B[u,v,w]_{B}]_{\mathcal {L}}+[Bu,Bv,Bl(\mathfrak {X})w]_{\mathcal {L}}\\{} & {} \quad -[Bu,Bv,[\mathfrak {X},Bw]_{\mathcal {L}}]_{\mathcal {L}}+[Bu,Bl(\mathfrak {X})v,Bw]_{\mathcal {L}}-[Bu,[\mathfrak {X},Bv]_{\mathcal {L}},Bw]_{\mathcal {L}}\\{} & {} \quad +[Bl(\mathfrak {X})u,Bv,Bw]_{\mathcal {L}}-[[\mathfrak {X},Bu]_{\mathcal {L}},Bv,Bw]_{\mathcal {L}}-B\big (m(Bu,Bl(\mathfrak {X})w)v\big )\\{} & {} \quad +B\big (m(Bu,[\mathfrak {X},Bw]_{\mathcal {L}})v\big )-B\big (r(Bv,Bl(\mathfrak {X})w)u\big )+B\big (r(Bv,[\mathfrak {X},Bw]_{\mathcal {L}})u\big )\\{} & {} \quad -B\big (l(Bu,Bl(\mathfrak {X})v)w\big )+B\big (l(Bu,[\mathfrak {X},Bv]_{\mathcal {L}})w\big )-B\big (r(Bl(\mathfrak {X})v,Bw)u\big )\\{} & {} \quad +B\big (r([\mathfrak {X},Bv]_{\mathcal {L}},Bw)u\big )-B\big (l(Bl(\mathfrak {X})u,Bv)w\big )+B\big (l([\mathfrak {X},Bu]_{\mathcal {L}},Bv)w\big )\\{} & {} \quad -B\big (m(Bl(\mathfrak {X})u,Bw)v\big )+B\big (m([\mathfrak {X},Bu]_{\mathcal {L}},Bw)v\big )\\{} & {} =-B\big (l(\mathfrak {X})l(Bu,Bv)w\big )-B\big (l(\mathfrak {X})m(Bu,Bw)v\big )-B\big (l(\mathfrak {X})r(Bv,Bw)u\big )\\{} & {} \quad +B\big (l(Bu,Bv)l(\mathfrak {X})w\big )+B\big (m(Bu,Bw)l(\mathfrak {X})v\big )+B\big (r(Bv,Bw)l(\mathfrak {X})u\big )\\{} & {} \quad +B\big (l(Bu,[\mathfrak {X},Bv]_{\mathcal {L}})w\big )+B\big (m(Bu,[\mathfrak {X},Bw]_{\mathcal {L}})v\big )+B\big (r(Bv,[\mathfrak {X},Bw]_{\mathcal {L}})u\big )\\{} & {} \quad +B\big (l([\mathfrak {X},Bu]_{\mathcal {L}},Bv)w\big )+B\big (m([\mathfrak {X},Bu]_{\mathcal {L}},Bw)v\big )+B\big (r([\mathfrak {X},Bv]_{\mathcal {L}},Bw)u\big )\\{} & {} =0. \end{aligned}$$

Thus, we deduce that $\partial _{B}\delta (\mathfrak {X})=0.$ $\square$

Now we give the cohomology of relative Rota-Baxter operators on a 3-Leibniz algebras $(\mathcal {L},$ $[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r).

Let B be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). Define the set of 0-cochains to be 0. For $n\ge 1,$ define the set of n-cochains by

$$\begin{aligned} \mathfrak {C}^{n}(B)= {\left\{ \begin{array}{ll} \mathfrak {C}_{\textsf {3Leib}}^{n-1}(V;\mathcal {L}),&{}n\ge 2,\\ \mathcal {L}\otimes \mathcal {L},&{} n=1. \end{array}\right. } \end{aligned}$$

Define $\textrm{d}_{\textsf {B}}:\mathfrak {C}^{n}(B)\rightarrow \mathfrak {C}^{n+1}(B)$ by $\textrm{d}_{\textsf {B}}= {\left\{ \begin{array}{ll} \partial _{B},&{} n\ge 2,\\ \delta ,&{} n=1. \end{array}\right. }$ Then $(\mathop {\oplus }\limits _{n=1}^{\infty } \mathfrak {C}^{n}(B),\textrm{d}_{\textsf {B}})$ is a cochain complex. Denote the set of n-cocycles by $\mathcal {Z}^{n}(B)$, the set of n-coboundaries by $\mathcal {B}^{n}(B)$ and the n-th cohomology group of the relative Rota-Baxter operator B by $\mathcal {H}^{n}(B)=\mathcal {Z}^{n}(B)/\mathcal {B}^{n}(B),~ n\ge 1.$

Definition 4.4

The cohomology of the cochain complex $(\mathop {\oplus }\limits _{n=1}^{\infty } \mathfrak {C}^{n}(B),\textrm{d}_{\textsf {B}})$ is taken to be the cohomology for the relative Rota-Baxter operator B.

Example 4.5

We consider the Rota-Baxter operator $B=\left( \begin{array}{ccc}0&{}0&{}1\\ 0&{}0&{}0\\ 0&{}0&{}1\end{array}\right)$ given in Example 2.11, and give its first and second cohomology group of the Rota-Baxter operator B respectively. By Proposition 2.13, we can get the 3-dimensional descendent 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{B})$ given with respect to a basis $\{e_1,e_2,e_3\}$ by

$$\begin{aligned}{}[e_3,e_2,e_3]_{B}=-e_2,\quad [e_3,e_3,e_2]_{B}=e_2. \end{aligned}$$

By (30)–(32), we obtain the regular representation $(\mathcal {L};l_{B},m_{B},r_{B})$ of the descendent 3-Leibniz algebra $\mathcal {L},$ where

$$\begin{aligned} \left\{ \begin{array}{rcl} l_{B}(e_3,e_3)(e_2)&{}=&{}e_2,\\ m_{B}(e_3,e_3)(e_2)&{}=&{}-e_2,\\ r_{B}(e_i,e_j)(e_k)&{}=&{}0,\quad \forall 1\le i,j,k \le 3. \end{array}\right. \end{aligned}$$

Consider the following cochains structure:

For all $\mathfrak {X}\in \otimes ^{2}\mathcal {L},$ let

$$\begin{aligned} \mathfrak {X}={}& {}\lambda _{11}e_1\otimes e_1+\lambda _{12}e_1\otimes e_2+\lambda _{13}e_1\otimes e_3+\lambda _{21}e_2\otimes e_1+\lambda _{22}e_2\otimes e_2 +\lambda _{23}e_2\otimes e_3\\{}& {}+\lambda _{31}e_3\otimes e_1+\lambda _{32}e_3\otimes e_2+\lambda _{33}e_3\otimes e_3. \end{aligned}$$

When $\delta (\mathfrak {X})=0,$ by (33), we have ${\left\{ \begin{array}{ll} \delta (\mathfrak {X})e_1=0,\\ \delta (\mathfrak {X})e_2=0,\\ \delta (\mathfrak {X})e_3=\lambda _{32}\delta (e_3\otimes e_2)e_3=\lambda _{32}e_2. \end{array}\right. }$ Since $e_2\ne 0,$ we have $\lambda _{32}=0.$ Thus,

$$\begin{aligned} \mathfrak {X}={}& {}\lambda _{11}e_1\otimes e_1+\lambda _{12}e_1\otimes e_2+\lambda _{13}e_1\otimes e_3+\lambda _{21}e_2\otimes e_1+\lambda _{22}e_2\otimes e_2 +\lambda _{23}e_2\otimes e_3\\{}& {}+\lambda _{31}e_3\otimes e_1+\lambda _{33}e_3\otimes e_3. \end{aligned}$$

We have $\mathcal {H}^{1}(B)\cong \mathbb {R}^8.$

For all $\theta \in \textrm{Hom}(\mathcal {L},\mathcal {L}),$ we get

$$\begin{aligned} \partial _{B}\theta (x,y,z)=-\theta ([x,y,z]_{B})+l_{B}(x,y)(\theta (z))+m_{B}(x,z)(\theta (y)). \end{aligned}$$

If 2-cochain $\partial _{B}$ is closed, that is $\partial _{B}\theta (e_i,e_j,e_k)=0$, where $1\le i,j,k \le 3,$ which implies that

$$\begin{aligned} \theta (e_2)= & {} \theta ([e_3,e_3,e_2]_{B})=l_{B}(e_3,e_3)(\theta (e_2)),\end{aligned}$$

(34)

$$\begin{aligned} -\theta (e_2)= & {} \theta ([e_3,e_2,e_3]_{B})=m_{B}(e_3,e_3)(\theta (e_2)). \end{aligned}$$

(35)

Suppose $\theta (e_2)=\lambda _{1}e_1+\lambda _{2}e_2+\lambda _{3}e_3.$ By (34) and (35), we have $\lambda _2=0.$ Then $\theta =\left( \begin{array}{ccc}b_{11}&{}b_{12}&{}b_{13}\\ b_{21}&{}0&{}b_{23}\\ b_{31}&{}b_{32}&{}b_{33}\end{array}\right) .$ By $\dim (\ker \delta )+\dim (\textrm{Im}\delta )=\dim (\otimes ^{2}\mathcal {L})=9,$ we have $\mathcal {H}^{2}(B)=\ker \partial _{B}/\textrm{Im}\delta \cong \mathbb {R}^{8}/\mathbb {R}=\mathbb {R}^{7}.$

At the end of this subsection, we give the relationship between the coboundary operator $\textrm{d}_{\textsf {B}}$ of the relative Rota-Baxter operator B and the differential $l_1^{B}$ defined by  (25) using the Maurer-Cartan element B of the Lie 3-algebra $(C_{\textsf {3Leib}}^*(V,\mathcal {L}),l_3)$.

Theorem 4.6

Let B be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ with respect to a representation (V; l, m, r). Then we have

$$\begin{aligned} \textrm{d}_{\textsf {B}} \theta =(-1)^{n-1}l_{1}^{B}\theta ,\forall \theta \in \textrm{Hom}(\underbrace{(\otimes ^{2}V)\otimes \cdots \otimes (\otimes ^{2}V)}_{n-1}\otimes V,\mathcal {L}), n\ge 1. \end{aligned}$$

(36)

Proof

For all $x,y,z \in \mathcal {L}, u,v,w \in V$, by Lemma 3.9, we have

$$\begin{aligned} {[}{[}\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}(u,v,w)= & {} 2[u,v,w]_{B},\\ {{[}{[}\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}(u,v,z)= & {} 2l_{B}(u,v)(z),\\ {{[}{[}\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}(u,y,w)= & {} 2m_{B}(u,w)(y),\\ {{[}{[}\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}(x,v,w)= & {} 2r_{B}(v,w)(x). \end{aligned}$$

In addition, for all $U_{i}=u_{i}\otimes v_{i}\in \otimes ^{2}V, i=1,2,\cdots ,n$ and $u_{n+1}\in V$, by (25), we get

$$\begin{aligned}{} & {} 2l_{1}^{B}\theta (U_{1},U_{2},\cdots ,U_{n},u_{n+1})\\{} & {} \quad =l_3(B,B,\theta )(U_{1},U_{2},\cdots ,U_{n},u_{n+1})\\{} & {} \quad ={[[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},\theta ]}_{\textsf {3Leib}}(U_{1},U_{2},\cdots ,U_{n},u_{n+1})\\{} & {} \quad ={[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}\big (\theta (U_{1},\cdots ,U_{n-1},u_{n})\otimes v_{n},u_{n+1}\big )\\{} & {} \qquad +{[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}\big (u_{n}\otimes \theta (U_{1},\cdots ,U_{n-1},v_{n}),u_{n+1}\big )\\{} & {} \qquad +\sum _{i=1}^{n}(-1)^{n-1}(-1)^{i-1}{[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}(U_{i},\theta (U_{1},\cdots ,\hat{U_{i}},\cdots ,U_{n},u_{n+1}))\\{} & {} \qquad -(-1)^{n-1}\sum _{k=1}^{n-1}\sum _{i=1}^{k}(-1)^{i+1}\theta \big (U_{1},\cdots ,\hat{U_{i}},\cdots ,U_{k},{[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}(U_{i},u_{k+1})\otimes v_{k+1},\\{} & {} U_{k+2},\cdots ,U_{n},u_{n+1}\big )\\{} & {} \qquad -(-1)^{n-1}\sum _{k=1}^{n-1}\sum _{i=1}^{k}(-1)^{i+1}\theta \big (U_{1},\cdots ,\hat{U_{i}},\cdots ,U_{k},u_{k+1}\otimes {[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}(U_{i},v_{k+1}),\\{} & {} U_{k+2},\cdots ,U_{n},u_{n+1}\big )\\{} & {} \qquad -(-1)^{n-1}\sum _{i=1}^{n}(-1)^{i+1}\theta \big (U_{1},\cdots ,\hat{U}_{i},\cdots ,U_{n},{[[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},B]_{\textsf {3Leib}},B]}_{\textsf {3Leib}}(U_{i},u_{n+1})\big )\\{} & {} \quad =2(-1)^{n-1}\textrm{d}_{\textsf {B}} \theta . \end{aligned}$$

Thus, we deduce that $\textrm{d}_{\textsf {B}} \theta =(-1)^{n-1}l_{1}^{B}\theta .$ $\square$

4.2 Infinitesimal Deformations of relative Rota-Baxter operators

In this subsection, we study the infinitesimal deformations of relative Rota-Baxter operators  on 3-Leibniz algebras using the established cohomology theory.

Let $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ be a 3-Leibniz algebra over $\mathbb {K}$ and $\mathbb {K}[t]$ be a polynomial ring in one variable t. Then $\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}$ is a $\mathbb {K}[t]/(t^2)$-module. Moreover, $\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}$ is a 3-Leibniz algebra over $\mathbb {K}[t]/(t^2)$, where the 3-Leibniz algebra structure is defined by

$$\begin{aligned} {[}f(t)\otimes _\mathbb {K}x,g(t)\otimes _\mathbb {K}y,h(t)\otimes _\mathbb {K}z]:=f(t)g(t)h(t)\otimes _\mathbb {K}[x,y,z]_{\mathcal {L}}, \end{aligned}$$

for all $f(t), g(t), h(t)\in {\mathbb {K}} [t]/(t^2),~x, y, z \in \mathcal {L}.$ Denote $f(t)\otimes _\mathbb {K}x$ by f(t)x. In the sequel, all the vector spaces are finite dimensional vector spaces over $\mathbb {K}.$

Definition 4.7

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ with respect to a representation (V; l, m, r). Let $\mathfrak {B}:V\rightarrow \mathcal {L}$ be a linear map. If $B_{t}=B+t\mathfrak {B}$ is still a relative Rota-Baxter operator modulo $t^2$, we say that $\mathfrak {B}$ generates an infinitesimal deformation of B.

Since $B_{t}=B+t\mathfrak {B}$ is a relative Rota-Baxter operator modulo $t^2$, for all $u,v,w\in V,$ consider the coefficients of t, we have

$$\begin{aligned}{} & {} {[}Bu,\mathfrak {B}v,Bw]_\mathcal {L}+[Bu,Bv,\mathfrak {B}w]_\mathcal {L}+[\mathfrak {B}u,Bv,Bw]_\mathcal {L}\nonumber \\{} & {} \quad =B\Big (l(Bu,\mathfrak {B}v)w+l(\mathfrak {B}u,Bv)w+m(Bu,\mathfrak {B}w)v+m(\mathfrak {B}u,Bw)v+r(Bv,\mathfrak {B}w)u+r(\mathfrak {B}v,Bw)u\Big )\nonumber \\{} & {} \qquad +\mathfrak {B}\Big (l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u\Big ), \end{aligned}$$

(37)

which means that $\mathfrak {B}$ is a 2-cocycle of the relative Rota-Baxter operator B. Hence, $\mathfrak {B}$ defines a cohomology class in $\mathcal {H}^{2}(B)$.

Definition 4.8

Let B and $B'$ be two relative Rota-Baxter operators on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L})$ with respect to a representation (V; l, m, r). A homomorphism from B to $B^{\prime }$ consists of a 3-Leibniz algebra homomorphism $\phi _{\mathcal {L}}:\mathcal {L}\rightarrow \mathcal {L}$ and a linear map $\phi _{V}:V\rightarrow V$ such that

$$\begin{aligned} B^{\prime }\circ \phi _{V}= & {} \phi _{\mathcal {L}}\circ B, \end{aligned}$$

(38)

$$\begin{aligned} \phi _{V}(l(x,y)u)= & {} l(\phi _{\mathcal {L}}(x),\phi _{\mathcal {L}}(y))\phi _{V}(u),\end{aligned}$$

(39)

$$\begin{aligned} \phi _{V}(m(x,y)u)= & {} m(\phi _{\mathcal {L}}(x),\phi _{\mathcal {L}}(y))\phi _{V}(u), \end{aligned}$$

(40)

$$\begin{aligned} \phi _{V}(r(x,y)u)= & {} r(\phi _{\mathcal {L}}(x),\phi _{\mathcal {L}}(y))\phi _{V}(u), \quad \forall x,y \in \mathcal {L}, u\in V. \end{aligned}$$

(41)

In particular, if both $\phi _{\mathcal {L}}$ and $\phi _{V}$ are invertible, $(\phi _{\mathcal {L}},\phi _{V})$ is called an isomorphism from B to $B^{\prime }$.

Definition 4.9

Let B be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ with respect to a representation (V; l, m, r). Two one-parameter infinitesimal deformations $B_{t}^{1}=B+t\mathfrak {B}_{1}$ and $B_{t}^{2}=B+t\mathfrak {B}_{2}$ are said to be equivalent if there exist $\mathfrak {X} \in \otimes ^{2}{\mathcal {L}}$, such that $(\textrm{Id}_{\mathcal {L}}+tL_{\mathfrak {X}}, \textrm{Id}_{V}+tl(\mathfrak {X}))$ is a homomorphism modulo $t^2$ from $B_{t}^{1}$ to $B_{t}^{2}$. In particular, an infinitesimal deformation $B_{t}^{1}=B+t\mathfrak {B}_{1}$ of a relative Rota-Baxter operator B is said to be trivial if there exist $\mathfrak {X} \in \otimes ^{2}{\mathcal {L}},$ such that $(\textrm{Id}_{\mathcal {L}}+tL_{\mathfrak {X}}, \textrm{Id}_{V}+tl(\mathfrak {X}))$ is a homomorphism modulo $t^2$ from $B_{t}^{1}$ to B.

Theorem 4.10

Let $B:V\rightarrow \mathcal {L}$ be a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathcal {L},[\cdot ,\cdot ,\cdot ]_{\mathcal {L}})$ with respect to a representation (V; l, m, r). If two one-parameter infinitesimal deformations $B_{t}^{1}=B+t\mathfrak {B}_{1}$ and $B_{t}^{2}=B+t\mathfrak {B}_{2}$ modulo $t^2$ are equivalent, then $\mathfrak {B}_{1}$ and $\mathfrak {B}_{2}$ are in the same cohomology class in $\mathcal {H}^{2}(B)$.

Proof

Let $(\textrm{Id}_{\mathcal {L}}+tL_{\mathfrak {X}}, \textrm{Id}_{V}+tl(\mathfrak {X}))$ be a homomorphism modulo $t^2$ from $B_{t}^{1}$ to $B_{t}^{2}$. By (38), we have

$$\begin{aligned} (\textrm{Id}_{\mathcal {L}}+tL_{\mathfrak {X}})(B+t\mathfrak {B}_{1})(u)=(B+t\mathfrak {B}_{2})(\textrm{Id}_{V}+tl(\mathfrak {X}))(u), \end{aligned}$$

which implies that

$$\begin{aligned} \mathfrak {B}_{1}(u)-\mathfrak {B}_{2}(u)=Bl(\mathfrak {X})(u)-[\mathfrak {X},Bu]_\mathcal {L}, \quad \forall \mathfrak {X} \in \otimes ^2\mathcal {L},~u \in V. \end{aligned}$$

Thus we have $\mathfrak {B}_{1}(u)-\mathfrak {B}_{2}(u)=(\textrm{d}_{\textsf {B}}\mathfrak {X})(u)$, that is, $\mathfrak {B}_{1}$ and $\mathfrak {B}_{2}$ are in the same cohomology class. $\square$

5 Relative Rota-Baxter 3-Leibniz Algebras

In this section, we construct an $L_{\infty }$-algebra that characterizes relative Rota-Baxter 3-Leibniz algebras as Maurer-Cartan elements. We establish the deformation theory of relative Rota-Baxter 3-Leibniz algebras. As applications, we use the second cohomology group to classify infinitesimal deformations of relative Rota-Baxter 3-Leibniz algebras.

5.1 The $L_{\infty }$-Algebra that Controls Deformations of Relative Rota-Baxter 3-Leibniz Algebras

Definition 5.1

A linear map $f\in \textrm{Hom}(\otimes ^p(\otimes ^{2}(\mathcal {L}_1\oplus \mathcal {L}_2))\otimes (\mathcal {L}_1\oplus \mathcal {L}_2),\mathcal {L}_1\oplus \mathcal {L}_2)$ has a bidegree m|n, which is denoted by $\Vert f\Vert =m|n$, if f satisfies the following four conditions:

• (i) $m+n=2p$;

• (ii) If X is an element in $\mathcal {L}^{m+1,n}$, then $f(X)\in \mathcal {L}_1$;

• (iii) If X is an element in $\mathcal {L}^{m,n+1}$, then $f(X)\in \mathcal {L}_2$;

• (iv) All the other cases, $f(X)=0$.

Denote the set of homogeneous linear maps of bidegree m|n by $C^{m\mid n}_\textsf {3Leib}(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2)$. It is obvious that we have the following lemmas:

Lemma 5.2

Let $f\in C_\textsf {3Leib}^p(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2)$ and $g\in C_\textsf {3Leib}^q(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2)$ be the homogeneous linear maps with bidegrees $m_f|n_f$ and $m_g|n_g$ respectively. Then $[f,g]_{\textsf {3Leib}}\in C_\textsf {3Leib}^{p+q}(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2)$ is a homogeneous linear map of bidegree $(m_f+m_g)|(n_f+n_g)$.

Proof

It is straightforward. $\square$

Lemma 5.3

If $||f||=-1|(2\,m+1)$ and $||g||=-1|(2n+1)$, then $[f,g]_{\textsf {3Leib}}=0.$ Consequently, $\oplus _{m=1}^{+\infty } C_{\textsf {3Leib}}^{-1|(2\,m+1)}(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2)$ is an abelian subalgebra of the graded Lie algebra $(C_{\textsf {3Leib}}^*(\mathcal {L}_1\oplus \mathcal {L}_2,\mathcal {L}_1\oplus \mathcal {L}_2),[\cdot ,\cdot ]_\textsf {3Leib}).$

Proof

It follows from Lemma 5.2. $\square$

Recall the desuspension operator $s^{-1}$ which changes the $\mathbb Z$-graded vector space $V=\oplus _{k\in {\mathbb {Z}}}V^k$ according to the rule $(s^{-1}V)^i:=V^{i+1}$. The degree $-1$ map $s^{-1}:V\,\rightarrow \,s^{-1}V$ is defined by sending $v\in V$ to its copy $s^{-1}v\in s^{-1}V$.

There is also an $L_\infty$-algebra structure on a bigger space, which is used to study simultaneous deformations of morphisms between Lie algebras in [5, 20].

Theorem 5.4

([36]) Let $(F,{\mathfrak {h}},\mathcal {P},\Delta )$ be a V-data. Then the graded vector space $s^{-1}F\oplus {\mathfrak {h}}$ is an $L_\infty$-algebra where

$$\begin{aligned} l_1(s^{-1}x,a)= & {} (-s^{-1}[\Delta ,x],\mathcal {P}(x+[\Delta ,a])),\\ l_2(s^{-1}x,s^{-1}y)= & {} (-1)^xs^{-1}[x,y],\\ l_k(s^{-1}x,a_1,\cdots ,a_{k-1})= & {} \mathcal {P}[\cdots [[x,a_1],a_2]\cdots ,a_{k-1}],\quad k\ge 2,\\ l_k(a_1,\cdots ,a_{k-1},a_k)= & {} \mathcal {P}[\cdots [[\Delta ,a_1],a_2]\cdots ,a_{k}],\quad k\ge 2. \end{aligned}$$

Here $a,a_1,\cdots ,a_k$ are homogeneous elements of ${\mathfrak {h}}$ and x, y are homogeneous elements of F. All the other $L_\infty$-algebra products that are not obtained from the ones written above by permutations of arguments, will vanish. $\square$

Proposition 5.5

([20]) Let $F'$ be a graded Lie subalgebra of F that satisfies $[\Delta ,F']\subset F'$. Then $s^{-1}F'\oplus {\mathfrak {h}}$ is an $L_\infty$-subalgebra of the above $L_\infty$-algebra $(s^{-1}F\oplus {\mathfrak {h}},\{l_k\}_{k=1}^{+\infty })$.

Let $\mathcal {L}$ and V be two vector spaces. According to the graded Lie algebra $(\oplus _{n=0}^{+\infty }C_\textsf {3Leib}^{n}(\mathcal {L}\oplus V,\mathcal {L}\oplus V),[\cdot ,\cdot ]_{\textsf {3Leib}})$, we can get a V-data structure, and an $L_\infty$-algebra naturally.

Proposition 5.6

We have a V-data $(F,{\mathfrak {h}},\mathcal {P},\Delta )$ as follows:

• the graded Lie algebra $(F,[\cdot ,\cdot ])$ is given by $\big (\oplus _{n=0}^{+\infty }C_\textsf {3Leib}^{n}(\mathcal {L}\oplus V,\mathcal {L}\oplus V),[\cdot ,\cdot ]_{\textsf {3Leib}}\big )$;

• the abelian graded Lie subalgebra ${\mathfrak {h}}$ is given by

  $$\begin{aligned} {\mathfrak {h}}:=\oplus _{n=0}^{+\infty }C_\textsf {3Leib}^{-1|(2n+1)}(\mathcal {L}\oplus V,\mathcal {L}\oplus V)=\oplus _{n=0}^{+\infty }\textrm{Hom}(\underbrace{(\otimes ^{2}V)\otimes \cdots \otimes (\otimes ^{2}V)}_{n}\otimes V,\mathcal {L}); \end{aligned}$$

  (42)

• $\mathcal {P}:F\,\rightarrow \,F$ is the projection onto the subspace ${\mathfrak {h}}$;

• $\Delta =0$.

Consequently, we obtain an $L_\infty$-algebra $(s^{-1}F\oplus {\mathfrak {h}},\{l_k\}_{k=1}^{+\infty })$, where $l_i$ are given by

$$\begin{aligned} l_1(s^{-1}Q,\theta )= & {} \mathcal {P}(Q),\\ l_2(s^{-1}Q,s^{-1}Q')= & {} (-1)^Qs^{-1}[Q,Q']_{\textsf {3Leib}}, \\ l_k(s^{-1}Q,\theta _1,\cdots ,\theta _{k-1})= & {} \mathcal {P}[\cdots [Q,\theta _1]_{\textsf {3Leib}},\cdots ,\theta _{k-1}]_{\textsf {3Leib}},\quad k\ge 2, \end{aligned}$$

for homogeneous elements $\theta ,\theta _1,\cdots ,\theta _{k-1}\in {\mathfrak {h}}$, homogeneous elements $Q,Q'\in F$ and all the other possible combinations vanish.

Proof

By Lemmas 5.2 and  5.3, ${\mathfrak {h}}=\oplus _{n=0}^{+\infty }C_{\textsf {3Leib}}^{-1|(2n+1)}(\mathcal {L}\oplus V,\mathcal {L}\oplus V)$ is an abelian subalgebra of $(F,[\cdot ,\cdot ])$. Since $\mathcal {P}$ is the projection onto ${\mathfrak {h}}$, it is obvious that $\mathcal {P}\circ \mathcal {P}=\mathcal {P}$. It is straightforward to deduce that the kernel of $\mathcal {P}$ is a graded Lie subalgebra of $(F,[\cdot ,\cdot ])$. Thus $(F,{\mathfrak {h}},\mathcal {P},\Delta =0)$ is a V-data. By Theorem 5.4, we can get the other conclusions immediately. $\square$

Denote by

$$\begin{aligned} C_{\textsf {3Leib}}^{2n|0}(\mathcal {L}\oplus V,\mathcal {L}\oplus V)=\textrm{Hom}(\otimes ^{n}(\otimes ^2\mathcal {L})\otimes \mathcal {L},\mathcal {L})\oplus \sum \limits _{i=0}^{2n}\textrm{Hom}(\otimes ^{i}\mathcal {L}\otimes V \otimes ( \otimes ^{2n-i}\mathcal {L}),V). \end{aligned}$$

By Lemma 5.2, we can deduce that

$$\begin{aligned} F'=\oplus _{n=0}^{+\infty }C_{\textsf {3Leib}}^{2n|0}(\mathcal {L}\oplus V,\mathcal {L}\oplus V), \end{aligned}$$

(43)

is a graded Lie subalgebra of $\big (\oplus _{n=0}^{+\infty }C_{\textsf {3Leib}}^{n}(\mathcal {L}\oplus V,\mathcal {L}\oplus V),[\cdot ,\cdot ]_{\textsf {3Leib}}\big )$. Furthermore, we have

Corollary 5.7

$(s^{-1}F'\oplus {\mathfrak {h}},\{l_i\}_{i=1}^{+\infty })$ is an $L_\infty$-algebra, where $l_i$ are given by

$$\begin{aligned} l_2(s^{-1}Q,s^{-1}Q')= & {} (-1)^Qs^{-1}[Q,Q']_{\textsf {3Leib}}, \\ l_k(s^{-1}Q,\theta _1,\ldots ,\theta _{k-1})= & {} \mathcal {P}[\ldots [Q,\theta _1]_{\textsf {3Leib}},\ldots ,\theta _{k-1}]_{\textsf {3Leib}},\quad k\ge 2, \end{aligned}$$

for homogeneous elements $\theta _1,\cdots ,\theta _{k-1}\in {\mathfrak {h}}$, homogeneous elements $Q,Q'\in F'$, and all the other possible combinations vanish.

Proof

It follows from Propositions 5.5 and  5.6. $\square$

Theorem 5.8

Let $\mathcal {L}$ and V be two vector spaces, $\mu \in \textrm{Hom}(\otimes ^3\mathcal {L},\mathcal {L}),~l,m,r\in \textrm{Hom}(\otimes ^{2}\mathcal {L}, \mathfrak {gl}(V))$ and $B\in \textrm{Hom}(V,\mathcal {L})$. Then $((\mathcal {L},\mu ),(V;l,m,r),B)$ is a relative Rota-Baxter 3-Leibniz algebra if and only if $(s^{-1}\pi ,B)$ is a Maurer-Cartan element of the $L_\infty$-algebra $(s^{-1}F'\oplus {\mathfrak {h}},\{l_i\}_{i=1}^{+\infty })$ given in Corollary 5.7, where $\pi =\hat{\mu }+\hat{l}+\hat{m}+\hat{r}\in C_{\textsf {3Leib}}^{2|0}(\mathcal {L}\oplus V,\mathcal {L}\oplus V)$ are given by (24).

Proof

Let $(s^{-1}\pi ,B)$ be a Maurer-Cartan element of $(s^{-1}F'\oplus {\mathfrak {h}},\{l_i\}_{i=1}^{+\infty })$. By Lemmas 5.2 and 5.3, we have

$$\begin{aligned} ||[\pi ,B]_{\textsf {3Leib}}||=1|1,&\quad ||[[\pi ,B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}||=0|2,\\ \quad ||[[[\pi ,B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}||&=-1|3,\quad [[[[\pi ,B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}=0. \end{aligned}$$

Then, by Corollary 5.7, we have

$$\begin{aligned}{} & {} \sum _{k=1}^{+\infty }\frac{1}{k!}l_k\Big ((s^{-1}\pi ,B),\cdots ,(s^{-1}\pi ,B)\Big )\\= & {} \frac{1}{2!}l_2\Big ((s^{-1}\pi ,B),(s^{-1}\pi ,B)\Big )+\frac{1}{3!}l_3\Big ((s^{-1}\pi ,B),(s^{-1}\pi ,B),(s^{-1}\pi ,B)\Big )\\{} & {} +\frac{1}{4!}l_4\Big ((s^{-1}\pi ,B),(s^{-1}\pi ,B),(s^{-1}\pi ,B),(s^{-1}\pi ,B)\Big )\\= & {} \Big (-s^{-1}\frac{1}{2}[\pi ,\pi ]_{\textsf {3Leib}},\frac{1}{3!}[[[\pi ,B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}\Big )\\= & {} (0,0). \end{aligned}$$

Thus, we obtain $~{[}\pi ,\pi ]_{\textsf {3Leib}}=0$ and ${[}[[\pi ,B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},B]_{\textsf {3Leib}}=0,$ which implies that $(\mathcal {L},\mu )$ is a 3-Leibniz algebra, (V; l, m, r) is its representation and B is a relative Rota-Baxter operator on the 3-Leibniz algebra $(\mathcal {L},\mu )$ with respect to the representation (V; l, m, r). Thus $((\mathcal {L},\mu ),(V;l,m,r),B)$ is a relative Rota-Baxter 3-Leibniz algebra if and only if $(s^{-1}\pi ,B)$ is a Maurer-Cartan element of the $L_\infty$-algebra $(s^{-1}F'\oplus {\mathfrak {h}},\{l_i\}_{i=1}^{+\infty })$. $\square$

Let $((\mathcal {L},\mu ),(V;l,m,r),B)$ be a relative Rota-Baxter 3-Leibniz algebra. Denote by $\pi =\hat{\mu }+\hat{l}+\hat{m}+\hat{r}\in C_{\textsf {3Leib}}^{2|0}(\mathcal {L}\oplus V,\mathcal {L}\oplus V)$. By Theorem 5.8, we obtain that $(s^{-1}\pi ,B)$ is a Maurer-Cartan element of the $L_\infty$-algebra $(s^{-1}F'\oplus {\mathfrak {h}},\{l_i\}_{i=1}^{+\infty })$ given in Corollary 5.7. Now we are ready to give the $L_\infty$-algebra that controls deformations of the relative Rota-Baxter 3-Leibniz algebra.

Theorem 5.9

With the above notations, we have the twisted $L_\infty$-algebra $\big (s^{-1}F'\oplus {\mathfrak {h}},\{l_k^{(s^{-1}\pi ,B)}\}_{k=1}^{+\infty }\big )$ associated to a relative Rota-Baxter 3-Leibniz algebra $((\mathcal {L},\mu ),(V;l,m,r),B)$, where $\pi =\hat{\mu }+\hat{l}+\hat{m}+\hat{r}$. Moreover, for linear maps $B'\in \textrm{Hom}(V,\mathcal {L})$, $\mu '\in \textrm{Hom}(\otimes ^3\mathcal {L},\mathcal {L})$ and $l',m',r'\in \textrm{Hom}(\otimes ^2\mathcal {L}, \mathfrak {gl}(V))$, the triple $((\mathcal {L},\mu +\mu '),(V;l+l',m+m',r+r'),B+B')$ is also a relative Rota-Baxter 3-Leibniz algebra if and only if $\big (s^{-1}(\hat{\mu '}+\hat{l'}+\hat{m'}+\hat{r'}),B'\big )$ is a Maurer-Cartan element of the twisted $L_\infty$-algebra $\big (s^{-1}F'\oplus {\mathfrak {h}},\{l_k^{(s^{-1}\pi ,B)}\}_{k=1}^{+\infty }\big )$.

Proof

If $((\mathcal {L},\mu +\mu '),(V;l+l',m+m',r+r'),B+B')$ is a relative Rota-Baxter 3-Leibniz algebra, then by Theorem 5.8, we deduce that $(s^{-1}(\hat{\mu }+\hat{\mu '}+\hat{l}+\hat{l'}+\hat{m}+\hat{m'}+\hat{r}+\hat{r'}),B+B')$ is a Maurer-Cartan element of the $L_\infty$-algebra given in Corollary 5.7. Moreover, by Lemma 3.4, we obtain that $(s^{-1}(\hat{\mu '}+\hat{l'}+\hat{m'}+\hat{r'}),B')$ is a Maurer-Cartan element of the $L_\infty$-algebra $\big (s^{-1}F'\oplus {\mathfrak {h}},\{l_k^{(s^{-1}\pi ,B)}\}_{k=1}^{+\infty }\big )$. $\square$

5.2 Cohomologies and Infinitesimal Deformations of Relative Rota-Baxter 3-Leibniz Algebras

We define the cohomology of relative Rota-Baxter 3-Leibniz algebras using the twisted $L_\infty$-algebra given in Theorem 5.9.

Lemma 5.10

$\big (s^{-1}F'\oplus {\mathfrak {h}},l_1^{(s^{-1}\pi ,B)}\big )$ is a complex, i.e. $l_1^{(s^{-1}\pi ,B)}\circ l_1^{(s^{-1}\pi ,B)}=0.$

Proof

Since $\big (s^{-1}F'\oplus {\mathfrak {h}},\{l_k^{(s^{-1}\pi ,B)}\}_{k=1}^{+\infty }\big )$ is an $L_\infty$-algebra, we have $l_1^{(s^{-1}\pi ,B)}\circ l_1^{(s^{-1}\pi ,B)}=0.$ $\square$

Let $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),(V;l,m,r),B)$ be a relative Rota-Baxter 3-Leibniz algebra. Set

$$\begin{aligned} {\mathfrak {C}}^n(\mathcal {L},(l,m,r))=\textrm{Hom}(\underbrace{(\otimes ^{2}\mathcal {L})\otimes \cdots \otimes (\otimes ^{2}\mathcal {L})}_{n-1}\otimes \mathcal {L},\mathcal {L})\oplus \sum \limits _{i=0}^{2n-2}\textrm{Hom}(\underbrace{\mathcal {L}\otimes \cdots \otimes \mathcal {L}}_{i}\otimes V\otimes \underbrace{\mathcal {L}\otimes \cdots \otimes \mathcal {L}}_{2n-i-2},V), \end{aligned}$$

and

$$\begin{aligned} {\mathfrak {C}}^n(B)=\textrm{Hom}(\underbrace{(\otimes ^{2}V)\otimes \cdots \otimes (\otimes ^{2}V)}_{n-2}\otimes V,\mathcal {L}). \end{aligned}$$

Define the set of 0-cochains $\mathfrak {\mathfrak {C}}^{0}(\mathcal {L},(l,m,r),B)$ to be 0,  and define the set of 1-cochains $\mathfrak {\mathfrak {C}}^{1}(\mathcal {L},$ (l, m, r), B) to be $\mathfrak {gl}({\mathfrak {g}})\oplus \mathfrak {gl}(V).$ For $n\ge 2,$ define the space of n-cochains ${\mathfrak {C}}^n(\mathcal {L},(l,m,r),B)$ to be

$$\begin{aligned} {\mathfrak {C}}^n(\mathcal {L},(l,m,r),B) :=&{\mathfrak {C}}^n(\mathcal {L},(l,m,r))\oplus {\mathfrak {C}}^n(B)={\mathfrak {C}}_{\textsf {3Leib}}^{2(n-1)|0}(\mathcal {L}\oplus V,\mathcal {L}\oplus V)\oplus {\mathfrak {C}}_{\textsf {3Leib}}^{(-1)|(2n-3)}(\mathcal {L}\oplus V,\mathcal {L}\oplus V)\\ =&\textrm{Hom}(\otimes ^{n-1}(\otimes ^2\mathcal {L})\otimes \mathcal {L},\mathcal {L})\oplus \sum \limits _{i=0}^{2n-2}\textrm{Hom}(\otimes ^{i}\mathcal {L}\otimes V\otimes (\otimes ^{2n-i-2}\mathcal {L}),V)\\&\oplus \textrm{Hom}(\otimes ^{n-2}(\otimes ^2 V)\otimes V,\mathcal {L}). \end{aligned}$$

Define the coboundary operator $\mathcal {D}:{\mathfrak {C}}^n(\mathcal {L},(l,m,r),B)\,\rightarrow \,{\mathfrak {C}}^{n+1}(\mathcal {L},(l,m,r),B)$ by

$$\begin{aligned} \begin{aligned} \mathcal {D}(f,\theta )&=(-1)^{n-2}\Big (-[\pi ,f]_{\textsf {3Leib}},[[[\pi ,B]_{\textsf {3Leib}},B]_\textsf {3Leib},\theta ]_{\textsf {3Leib}}\\&\quad +\frac{1}{(2n-1)!}\underbrace{[\cdots [[}_{2n-1}f,B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},\cdots ,B]_{\textsf {3Leib}}\Big ),\\ \end{aligned} \end{aligned}$$

(44)

where $f\in \textrm{Hom}(\underbrace{(\otimes ^{2}\mathcal {L})\otimes \cdots \otimes (\otimes ^{2}\mathcal {L})}_{n-1}\otimes \mathcal {L},\mathcal {L})\oplus \sum \limits _{i=0}^{2n-2}\textrm{Hom}(\underbrace{\mathcal {L}\otimes \cdots \otimes \mathcal {L}}_{i}\otimes V\otimes \underbrace{\mathcal {L}\otimes \cdots \otimes \mathcal {L}}_{2n-i-2},V))$ and $\theta \in \textrm{Hom}(\underbrace{(\otimes ^{2}V)\otimes \cdots \otimes (\otimes ^{2}V)}_{n-2}\otimes V,\mathcal {L}).$

Theorem 5.11

With the above notations, $(\oplus _{n=0}^{+\infty }{\mathfrak {C}}^n(\mathcal {L},(l,m,r),B),\mathcal {D})$ is a cochain complex, i.e. $\mathcal {D}\circ \mathcal {D}=0.$

Proof

For any $(f,\theta )\in {\mathfrak {C}}^n(\mathcal {L},(l,m,r),B)$, we have $(s^{-1}f,\theta )\in (s^{-1}F'\oplus {\mathfrak {h}})^{n-2}$. By (44), we deduce that

$$\begin{aligned} \mathcal {D}(f,\theta )=(-1)^{n-2} l_1^{(s^{-1}\pi ,B)}(s^{-1}f,\theta ). \end{aligned}$$

By Lemma 5.10, we obtain that $(\oplus _{n=0}^{+\infty }{\mathfrak {C}}^n(\mathcal {L},(l,m,r),B),\mathcal {D})$ is a cochain complex. $\square$

Definition 5.12

The cohomology of the cochain complex $(\oplus _{n=0}^{+\infty }{\mathfrak {C}}^n(\mathcal {L},(l,m,r),B),\mathcal {D})$ is called the cohomology of the relative Rota-Baxter 3-Leibniz algebra $((\mathcal {L},\mu ),(V;l,m,r),B)$. Denote its n-th cohomology group by $\mathcal {H}^n(\mathcal {L},(l,m,r),B).$

For any $f\in {\mathfrak {C}}^n(\mathcal {L},(l,m,r))$, define an operator $\hat{\partial }:{\mathfrak {C}}^n(\mathcal {L},(l,m,r))\,\rightarrow \,{\mathfrak {C}}^{n+1}(\mathcal {L},(l,m,r))$ by

$$\begin{aligned} \hat{\partial } f:=(-1)^{n-1}[\hat{\mu }+\hat{l}+\hat{m}+\hat{r},f]_{\textsf {3Leib}}. \end{aligned}$$

(45)

Define a linear operator $h_B:{\mathfrak {C}}^n(\mathcal {L},(l,m,r))\,\rightarrow \,{\mathfrak {C}}^{n+1}(B)$ by

$$\begin{aligned} h_Bf:=(-1)^{n-2} \frac{1}{(2n-1)!}\underbrace{[\cdots [[}_{2n-1}f,B]_{\textsf {3Leib}},B]_{\textsf {3Leib}},\cdots ,B]_{\textsf {3Leib}}. \end{aligned}$$

(46)

Thus, we have

$$\begin{aligned} \mathcal {D}(f,\theta )=(\hat{\partial } f,\textrm{d}_{\textsf {B}}\theta +h_Bf). \end{aligned}$$

(47)

Furthermore, we obtain the following diagram

Finally, we give the application of the cohomology theory of relative Rota-Baxter 3-Leibniz algebras.

Definition 5.13

Let $((\mathcal {L},\mu ),(V;l,m,r),B)$ be a relative Rota-Baxter 3-Leibniz algebra and $\omega _1\in \textrm{Hom}(\otimes ^{3}\mathcal {L},\mathcal {L}), \mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1\in \textrm{Hom}(\otimes ^2\mathcal {L}, \mathfrak {gl}(V)), \mathcal {B}_1\in \textrm{Hom}(V,\mathcal {L})$ be linear maps. Consider a t-paramater family of trilinear operations

$$\begin{aligned} {[}x,y,z]_t=[x,y,z]_\mathcal {L}+t\omega _1(x,y,z),\quad \forall x,y,z\in \mathcal {L}. \end{aligned}$$

If the 3-brackets $[\cdot ,\cdot ,\cdot ]_t$ are 3-Leibniz algebra structures on $\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}$, $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K}V;l+t\mathfrak {l}_1,m+t\mathfrak {m}_1,r+t\mathfrak {r}_1)$ is a representation of $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}, [\cdot ,\cdot ,\cdot ]_t)$, and $B_t=B+t\mathcal {B}_1$ is still a relative Rota-Baxter operator on the 3-Leibniz algebra $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}, [\cdot ,\cdot ,\cdot ]_t)$, we say that $(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)$ generates an infinitesimal deformation of the relative Rota-Baxter 3-Leibniz algebra.

Since $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}, [\cdot ,\cdot ,\cdot ]_t)$ is a 3-Leibniz algebra, we have

$$\begin{aligned}{} & {} \omega _1(x_1,x_2,[x_3,x_4,x_5]_\mathcal {L})+[x_1,x_2,\omega _1(x_3,x_4,x_5)]_\mathcal {L}\nonumber \\{} & {} \quad =[\omega _1(x_1,x_2,x_3),x_4,x_5]_\mathcal {L}+\omega _1([x_1,x_2,x_3]_\mathcal {L},x_4,x_5)+[x_3,\omega _1(x_1,x_2,x_4),x_5]_\mathcal {L}\nonumber \\{} & {} \qquad +\omega _1(x_3,[x_1,x_2,x_4]_\mathcal {L},x_5)+[x_3,x_4,\omega _1(x_1,x_2,x_5)]_\mathcal {L}+\omega _1(x_3,x_4,[x_1,x_2,x_5]_\mathcal {L}), \end{aligned}$$

(48)

which implies that $\partial \omega _1=0$.

Since $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} V;l+t\mathfrak {l}_1,m+t\mathfrak {m}_1,r+t\mathfrak {r}_1)$ is a representation of $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}, [\cdot ,\cdot ,\cdot ]_t)$, we obtain

$$\begin{aligned}{} & {} \mathfrak {l}_1(x_1,x_2)l(x_3,x_4)+l(x_1,x_2)\mathfrak {l}_1(x_3,x_4)\nonumber \\{} & {} \quad =l(\omega _1(x_1,x_2,x_3),x_4)+\mathfrak {l}_1([x_1,x_2,x_3]_\mathcal {L},x_4)+\mathfrak {l}_1(x_3,[x_1,x_2,x_4]_\mathcal {L})\nonumber \\{} & {} \qquad +l(x_3,\omega _1(x_1,x_2,x_4))+\mathfrak {l}_1(x_3,x_4)l(x_1,x_2)+l(x_3,x_4)\mathfrak {l}_1(x_1,x_2), \end{aligned}$$

(49)

$$\begin{aligned}{} & {} \mathfrak {l}_1(x_1,x_2)m(x_3,x_4)+l(x_1,x_2)\mathfrak {m}_1(x_3,x_4)\nonumber \\{} & {} \quad =m(\omega _1(x_1,x_2,x_3),x_4)+\mathfrak {m}_1([x_1,x_2,x_3]_\mathcal {L},x_4)+\mathfrak {m}_1(x_3,[x_1,x_2,x_4]_\mathcal {L})\nonumber \\{} & {} \qquad +m(x_3,\omega _1(x_1,x_2,x_4))+\mathfrak {m}_1(x_3,x_4)l(x_1,x_2)+m(x_3,x_4)\mathfrak {l}_1(x_1,x_2), \end{aligned}$$

(50)

$$\begin{aligned}{} & {} \mathfrak {l}_1(x_1,x_2)r(x_3,x_4)+l(x_1,x_2)\mathfrak {r}_1(x_3,x_4)\nonumber \\{} & {} \quad =r(\omega _1(x_1,x_2,x_3),x_4)+\mathfrak {r}_1([x_1,x_2,x_3]_\mathcal {L},x_4)+\mathfrak {r}_1(x_3,[x_1,x_2,x_4]_\mathcal {L})\nonumber \\{} & {} \qquad +r(x_3,\omega _1(x_1,x_2,x_4))+\mathfrak {r}_1(x_3,x_4)l(x_1,x_2)+r(x_3,x_4)\mathfrak {l}_1(x_1,x_2), \end{aligned}$$

(51)

$$\begin{aligned}{} & {} \mathfrak {m}_1(x_1,[x_2,x_3,x_4]_\mathcal {L})+m(x_1,\omega _1(x_2,x_3,x_4))\nonumber \\{} & {} \quad =r(x_3,x_4)\mathfrak {m}_1(x_1,x_2)+\mathfrak {r}_1(x_3,x_4)m(x_1,x_2)+m(x_2,x_4)\mathfrak {m}_1(x_1,x_3)\nonumber \\{} & {} \qquad +\mathfrak {m}_1(x_2,x_4)m(x_1,x_3)+l(x_2,x_3)\mathfrak {m}_1(x_1,x_4)+\mathfrak {l}_1(x_2,x_3)m(x_1,x_4), \end{aligned}$$

(52)

$$\begin{aligned}{} & {} \mathfrak {r}_1(x_1,[x_2,x_3,x_4]_\mathcal {L})+r(x_1,\omega _1(x_2,x_3,x_4))\nonumber \\{} & {} \quad =r(x_3,x_4)\mathfrak {r}_1(x_1,x_2)+\mathfrak {r}_1(x_3,x_4)r(x_1,x_2)+m(x_2,x_4)\mathfrak {r}_1(x_1,x_3)\nonumber \\{} & {} \qquad +\mathfrak {m}_1(x_2,x_4)r(x_1,x_3)+l(x_2,x_3)\mathfrak {r}_1(x_1,x_4)+\mathfrak {l}_1(x_2,x_3)r(x_1,x_4). \end{aligned}$$

(53)

Finally, since $B+t\mathcal {B}_1$ is a relative Rota-Baxter operator on a 3-Leibniz algebra $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} \mathcal {L}, [\cdot ,\cdot ,\cdot ]_t)$ with respect to a representation $(\mathbb {K}[t]/(t^2)\otimes _\mathbb {K} V;l+t\mathfrak {l}_1,m+t\mathfrak {m}_1,r+t\mathfrak {r}_1)$, we get

$$\begin{aligned}{} & {} [Bu,Bv,\mathcal {B}_1w]_\mathcal {L}+[Bu,\mathcal {B}_1v,Bw]_\mathcal {L}+[\mathcal {B}_1u,Bv,Bw]_\mathcal {L}+\omega _1(Bu,Bv,Bw)\nonumber \\{} & {} \quad =\mathcal {B}_1\big (l(Bu,Bv)w+m(Bu,Bw)v+r(Bv,Bw)u\big )+B\Big (\mathfrak {l}_1(Bu,Bv)w+l(\mathcal {B}_1u,Bv)w+l(Bu,\mathcal {B}_1v)w\nonumber \\{} & {} \qquad +\mathfrak {m}_1(Bu,Bw)v+m(\mathcal {B}_1u,Bw)v+m(Bu,\mathcal {B}_1w)v+\mathfrak {r}_1(Bv,Bw)u+r(\mathcal {B}_1v,Bw)u+r(Bv,\mathcal {B}_1w)u\Big ). \end{aligned}$$

(54)

Proposition 5.14

The triple $(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)$ determines an infinitesimal deformation of a relative Rota-Baxter 3-Leibniz algebra $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),(V;l,m,r),B)$ if and only if $(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)$ is a 2-cocycle of the relative Rota-Baxter 3-Leibniz algebra $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),(V;l,m,r),B)$.

Proof

By (48)–(54), $(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)$ is a 2-cocycle if and only if $(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)$ determines an infinitesimal deformation of the relative Rota-Baxter 3-Leibniz algebra $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),(V;$ l, m, r), B). $\square$

Definition 5.15

Let $((\mathcal {L},\mu ),(V;l,m,r),B)$ be a relative Rota-Baxter 3-Leibniz algebra. Two infinitesimal deformations $([\cdot ,\cdot ,\cdot ]'_t,l+t\mathfrak {l}'_1,m+t\mathfrak {m}'_1,r+t\mathfrak {r}'_1,B+t\mathcal {B}'_1)$ and $([\cdot ,\cdot ,\cdot ]_t,l+t\mathfrak {l}_1,m+t\mathfrak {m}_1,r+t\mathfrak {r}_1,B+t\mathcal {B}_1)$ generated by $(\omega '_1,(\mathfrak {l}'_1,\mathfrak {m}'_1,\mathfrak {r}'_1),\mathcal {B}'_1)$ and $(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)$ are said to be equivalent if there exist linear maps $N\in \textrm{Hom}(\mathcal {L},\mathcal {L}), S\in \textrm{Hom}(V,V)$ such that the $\mathbb {K}[t]/(t^2)$-module map $\Psi _t=\textrm{Id}_\mathcal {L}+tN, \Phi _t=\textrm{Id}_V+tS$ satisfy the following conditions:

$$\begin{aligned} \Psi _t([x,y,z]'_t)= & {} [\Psi _t(x),\Psi _t(y),\Psi _t(z)]_t; \end{aligned}$$

(55)

$$\begin{aligned} \mathcal {B}_t\circ \Phi _t= & {} \Psi _t\circ \mathcal {B}'_t;\end{aligned}$$

(56)

$$\begin{aligned} \Phi _t((l+t\mathfrak {l}'_1)(x,y)u)= & {} (l+t\mathfrak {l}_1)(\Psi _t(x),\Psi _t(y))(\Phi _t(u));\end{aligned}$$

(57)

$$\begin{aligned} \Phi _t((m+t\mathfrak {m}'_1)(x,y)u)= & {} (m+t\mathfrak {m}_1)(\Psi _t(x),\Psi _t(y))(\Phi _t(u));\end{aligned}$$

(58)

$$\begin{aligned} \Phi _t((r+t\mathfrak {r}'_1)(x,y)u)= & {} (r+t\mathfrak {r}_1)(\Psi _t(x),\Psi _t(y))(\Phi _t(u)). \end{aligned}$$

(59)

Now we give the important result in this subsection.

Theorem 5.16

There is a one-to-one correspondence between equivalence classes of infinitesimal deformations of the relative Rota-Baxter 3-Leibniz algebra $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),(V;l,m,r),B)$ and the second cohomology group $\mathcal {H}^2(\mathcal {L},(l,m,r),B)$.

Proof

By (55), we have

$$\begin{aligned} \omega '_1-\omega _1=\partial N. \end{aligned}$$

(60)

By (56), we get

$$\begin{aligned} \mathcal {B}'_1-\mathcal {B}_1=B\circ S-N\circ B. \end{aligned}$$

(61)

By (57)–(59), we obtain

$$\begin{aligned} \mathfrak {l}'_1(x,y)u-\mathfrak {l}_1(x,y)u= & {} l(x,y)(Su)+l(x,Ny)u+l(Nx,y)u-S(l(x,y)u); \end{aligned}$$

(62)

$$\begin{aligned} \mathfrak {m}'_1(x,y)u-\mathfrak {m}_1(x,y)u= & {} m(x,y)(Su)+m(x,Ny)u+m(Nx,y)u-S(m(x,y)u);\end{aligned}$$

(63)

$$\begin{aligned} \mathfrak {r}'_1(x,y)u-\mathfrak {r}_1(x,y)u= & {} r(x,y)(Su)+r(x,Ny)u+r(Nx,y)u-S(r(x,y)u). \end{aligned}$$

(64)

Therefore, we deduce that

$$\begin{aligned} (\omega _1',(\mathfrak {l}_1',\mathfrak {m}_1',\mathfrak {r}_1'),\mathcal {B}_1')-(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)=\mathcal {D}(N,S), \end{aligned}$$

which implies that $(\omega _1,(\mathfrak {l}_1,\mathfrak {m}_1,\mathfrak {r}_1),\mathcal {B}_1)$ and $(\omega _1',(\mathfrak {l}_1',\mathfrak {m}_1',\mathfrak {r}_1'),\mathcal {B}_1')$ are in the same cohomology class if and only if the corresponding infinitesimal deformations of $((\mathcal {L},[\cdot ,\cdot ,\cdot ]_\mathcal {L}),(V;l,m,r),B)$ are equivalent. $\square$